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INTRODUCTION 



The purpose of this final report is to present Battelle 
Columbus Laboratories ' principal findings for the 0E0 Experiment in 
Educational Performance Contracting. This report represents an inte- 
gration of analysis results presented in Battelle* s Interim Report* 
with results from additional analyses performed since that time. 

This report is divided into eight sections: 

(1) Description of The Experiment 

(2) Description of The Technology Company Programs 

(3) Description of the Target Population 

(4) Achievement Measures 

(5) Testing Procedures and Conditions 

(6) The Data-Analysis Method 

(7) Results and Conclusions: Technology Company Sites 

(8) Results and Conclusions: Incentive Only Sites. 

In section one (Description of Experiment) , the goal of the 
experiment is outlined, along with a summarization of the more important 
characteristics of the experimental design and experimental procedures. 

Section two provides an overview of the technology company 
programs, in terms of personnel utilized by the companies, curriculum 
and materials, and the incentive systems and strategies used. 

Section three presents descriptive information on some of the 
more important variables characterizing the socioeconomic status, social 
and home background, and achievement levels of students who participated 
in the experiment, so as to afford the reader a description of the type 
of student for which the results and conclusions of this study apply. 

In section four (Achievement Measures) , a summary is given 
of the criteria employed for the selection of achievement tests used 
to assess program impact on student learning, along with a brief 



* Interim Report on The Office of Economic Opportunity Experiment in 
Educational Performance Contracting, January 29, 1972. Battelle, 
Columbus Laboratories, 505 King Avenue, Columbus, Ohio, 43201. 
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"'escription of the tests selected. Section five, then, outlines the 
design and procedures " ?d to adminster the selected tests, followed 
\y e summary of the testing conditions for both the pre- and posttesting 
pnases . 

Section six outlines the method used for the analysis oi 
student achievement test data, and the rationale for the analysis 
method employed. This section will be of most benefit to a reader 
with basic training in statistical methodology. However, section six 
will also provide the non-statistically trained reader with a general 
orientation to the analysis methodology employed and the rationale 
for the use of this method. 

Section seven of the report presents the principal results 
and conclusions concerning the impact of educational performance 
contracting on student achievement in the eighteen school districts 
that contracted with private educational technology companies. In the 
last section of the report, results and conclusions are presented 
for the two "Incentives Only" sites (Stockton, California, and Mesa, 
Arizona), where performance incentive contracting was implemented 
without the use of a private educational technology company. 

Sections one, two, four, and five as outlined above contain 
essentially the same contents as were in Battelle’s Interim Report. 

Section three (Description of the Target Population) is expanded, 
providing additional information on experimental and control groups 
of students and differences in the characteristics of these two 
groups. Sections six, seven, and eight are changed, reflecting additional 
tasks performed since the Interim Report. 

Several Appendices to the report are provided that contain 
basic statistical data, so as to afford the interested reader an 
opportunity to examine and analyze detailed data hot always presented 
in the body of. the report, but which serve as a basis for the results 
and conclusions presented. 
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DESCRIPTION OF THE EXPERIMENT 

In view of the inability of traditional educational methods to 
narrow the gap between the academic performance of low- income, under- 
achieving students and "average" students, and the increasing public 
concern for accountability in education, 0E0 implemented an experimental 
remedial education program in reading and mathematics during the 
school year 1970-1971. This program involved federal support to 
participating school districts for the subcontracting of remedial 
teaching in reading and mathematics to private educational technology 
companies. These private companies subcontracted with the schools on a 
performance incentive basis. That is, the companies were to be remunerated 
for their services in proportion to the achievement gains of students 
throughout the year. 

Contracts were signed with two organizations to provide manage- 
ment support for the implementation of these subcontracts, to provide 
economic analyses of the costs of the programs, and to provide an evalua- 
tion of the educational achievement benefits obtained from each of the 
programs. Battelle has performed the evaluation of the educational 
achievement in the experiment. This section of the report includes a des- 
cription of all facets of the experiment in performance-incentive remedial 
education which are relevant to the analysis and evaluation of its educa- 
tional outcomes, based upon testing during school years 1970-71 and 1971-72. 

The general goal of this experiment was to determine how effec- 
tive the innovative programs would be in producing significant gains in 
the reading and mathematics skills of low- achieving students from low- 
income families. A general hypothesis that could be stated concerning 
the experimental programs was that low income, low-achieving students 
receiving instruction in the experimental programs for 1 year in reading 
and mathematics would show superior achievement - measured by standardized 
tests to that of similar students receiving normal school instruction 
in the same subjects. 

This experiment can be characterized as an effort to apply the 
concepts and methodology of laboratory experimentation in a real-world 
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setting, in order to test the research hypothesis. Clearly, the degree of 
experimental "control" possible in a real-world setting falls far short of 
that obtainable in a pure laboratory setting. On the other hand, the 
experiment represents much more than the unstructured collection of observa- 
tional data in order to answer the main research question. Indeed, con- 
siderable efforts were extended to achieve as much structure and control 
as is possible in a real-world environment, in order to provide a valid 
test of the research hypothesis. 

As with most field experiments, compromises had to be made in 
the experimental design which made the testing of the research hypothesis 
difficult. The most notable was the nonrandom assignment of students 
into the experimental and regular school classes. Random assignment of 
students was impossible, given the size of the experiment. However, 
efforts were made to select students for experimental and for regular 
instruction in a way so as to achieve a "match" between the two groups of 
students on several important variables affecting student performance. 

Thus, efforts were made to achieve comparable groups with respect to 
entry level achievement of the students, and with respect to family back- 
ground variables reflecting the socioeconomic status of students (e.g. , 
family income level). Further, the design sought to measure several of 
the pertinent variables affecting student performance, in order to 
determine the comparability of students receiving experimental and regular 
instruction, and for the purpose of taking into account in the analysis 
group differences that might exist. 

In order to accomplish the overall goal of the experiment, the 
following experimental features were adopted by 0E0: 

(1) A large number of school districts, varying in size, 
geographical location, rural-urban setting, and minority 
group composition were selected to participate, in order 

to provide a comprehensive test of the experimental programs. 

(2) Programs were implemented in the. same grades in all school 
districts for the same number of student;'. Grades 1, 2, 

3 and 7, 8, and 9 were selected with enrollment generally 
100 students per grade. In total, some 13,000 students 
participated ir. experimental programs and 12,000 in control 
programs. 
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(3) The performance of students in the experimental programs 
was compared with that of "control" students within the 
same school district receiving normal school instruc- 
tional programs . 

(4) Experimental and control schools were selected within 
districts with the objective of obtaining schools whose 
student populations were matched on achievement 
deficiency, family income, and minority group composition, 

(5) Sufficient numbers of students vrere to be selected to 
enable the study of student characteristics which might 
be related to student performance. 

(6) Technology companies (subcontractors) were selected to 
represent a variety of educational technology and 
approaches . 

Selection of School Districts 

From a pool of over 200 school districts that had expressed 
interest in the 0E0 performance -incentive remedial education experiment, 
163 replied to an 0E0 invitation to participate in the experiment. Of 
these, 77 made formal application to participate. In order to select 
school districts, 0E0 applied the following criteria; 

(1) Students had to come from '’low-income' 1 families, 
according to ESEA Title I or 0E0 criteria. 

(2) There had to be at least 100 students in each of 
grades 1,2, 3, 7, 8, and 9 deficient in reading and 
mathematics achievement who could serve as an 
experimental population and an equal number to serve 
as a control population. 

(3) There were recent (1968-69 or 1969-70) valid and 
reliable achievement test data available to facilitate 
the assessment of student populations within the 
schools of the district who could serve as experi- 
mental and control groups. 

(4) The school district had some familiarity with performance 
contracting . 

(5) The school district showed no evidence of having political, 
social, or economic problems that might interfere with 

the implementation of the experiment. 
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Of the 77 school districts that applied, 18 were selected for participation, 
as listed below. 



« Anchorage, Alaska 
® Athens, Georgia 

o Bronx, New York 
9 Dallas, Texas 
9 Fresno, California 
® Grand Rapids, Michigan 



Hartford, Connecticut 
Jacksonville, Florida 
Las Vegas, Nevada 
® McComb, Mississippi 
9 Philadelphia, Pennsylvania 



Seattle, Washington 
Selmer, Tennessee 
Taft, Texas 
Wichita, Kansas 



The districts were selected to be representative of the many different 
types that contain low-income, low-achieving populations in the United States. 
As can be seen the 18 districts included 4 large urban school systems 
(Bronx, Philadelphia, Dallas, and Seattle), plus several middle-size urban 
systems. Smaller and rural systems were represented by Athens, McComb, 
Rockland, Selmer, and Taft. 



Selection of Technology Companies 



Thirty-one technology companies applied to participate in the 
experiment in response to a competitive request for proposals issued by 0E0. 
Six firms were selected. The instructional approaches of these firms 
varied by relative emphasis upon hardware, incentives, or curricular soft- 
ware and teacher training methods. The firms were 

(1) Alpha Learning Systems, Inc. 

(2) Singer/Graf lex, Inc. 

(3) Westinghouse Learning Corporation 

(4) Quality Educational Development, Inc. 

(5) Learning Foundations, Inc. 

(6) Plan Education Centers, Inc. 

An attempt was made to have each company instruct as many different sub- 
populations of students (e.; ., white, black, Spanish-speaking, etc.) as 
possible. Consequently, each company was assigned to work in widely 
varying districts. With 18 districts in total, each of the 6 companies 
was assigned to work in 3 districts. 

These 6 companies provided a range of educational approaches. 

Their diversity is described elsewhere in this report (Reference Section 
on Description of Experimental Programs). 



O 
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Selection of Schools and the Creatio n 
of Experimental and Control Groups 

Within each selected school district, an effort was made to 
identify schools which could provide 100 students in each of six grades 
(1,2,3, 7,8 and 9) for the experimental groups'. In three of the districts, 
this requirement was reduced to 75 per grade, so as to include smaller 
districts fn the study. Students had to be deficient in reading and 
mathematics skills, from low- income families, and be representative of 
minority groups within the district. Usually at the primary level, more 
than two schools had to be selected to provide the 100 students in each 
of grades 1, 2 and 3. This was because of the normal small size of such 
schools. At the secondary level, one school frequently could be selected, 
although in some cases .ore than one was necessary to provide the 
designated experimental populations. 

The basic design called for a comparison of experimental (E) 
students receiving the innovative instruction with control (C) students 
receiving normal instruction in other schools within the same districts. 

The control students were designated in different school buildings in order 
to prevent any confounding of E versus C comparisons by having the effect 
of the experimental programs "rub off" or become assimilated through 
adjacent classrooms. Control schools, too, had to have a population of 
100 low-achieving students from low-income families in each of grades 1, 

2,3 and 7,8 and 9 . 

In order to measure any "rub off” effect in the experimental 
schools, an additional 50 students in each of the six grades were desig- 
nated as "comparison" students. These students were tested with the 
experimental students and were also to serve as a replacement pool for 
students who might leave the experimental programs' during the year. 

A fourth group was also established in some districts. Where 
the district already was operating another special reading and math program, 
0E0 offered to test those students and compare them with other groups in 
the experiment. These were known as "Special Treatment" groups. They 
were found in Grand Rapids, where two other non-OEO performance contracts 
and one special school project were operated, and in Hartford. 
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Since the comparison of the performance of E and C students was 
of primary interest, the selection of individual schools within a district 
focused on these populations. Criteria employed for the selection of E 
and C schools were 

(1) The schools had to be ESEA Title I designees, or, 
if not, at least 80% of the students to be in the 
project had to meet at least the Title I criteria. 

(2) The schools had to be among the most academically 
deficient Title I schools in the district. 

(3) There could be no other "special programs" in reading 
and mathematics within the particular schools that 
could affect the progress of groups in the experiment. 

(4) As stated, E schools had to contain 100 academically 
deficient students per grade (75 students per grade in 
three predominantly rural districts) and at least one- 
half again as many academically deficient students per 
grade who would be available as replacement students 
(the comparison group). In C schools, 100 academically 
deficient students per grade (75 students per grade in 
three predominantly rural districts) had to be present. 

The process of selecting schools as experimental and control 
within a district was based upon an analysis of their relative deficiencies 
in reading and math as shown by the districts' testing data. These were 
the data required to be present for a district to be considered for inclu- 
sion in the experiment. During the summer, a subcontractor to the manage- 
ment support contractor arrayed the district test data to show the relative 
ranking of Title I eligible schools in reading and math achievement. 

Generally speaking, the most deficient school or schools were selected as 
the experimental school(s) and the next most deficient as the control 
schools. In large districts where there was sometimes a substantial choice 
of schools, the selection was affected by the presence or absence of other 
special programs in them, the receptivity of individual schools to being 
included in the. experiment and the relative efficiency with which the 
required number of experimental and control students could be accumulated 
in them. For two of the smaller districts, Rockland and Taft, the control 
schools had to be selected in an adjoining district. 




zz 



9 



Selection of Students 

Once experimental and control schools were selected, students 
within these schools were selected into appropriate groups in a similar 
manner. The management support contractor arrayed reading and math scores 
of individual students by school to show the lowest achieving first, the 
next lowest achieving, etc. The student scores were shown in terms of 
decrements below grade level, i.e., the number of grade levels (in 
tenths) each student was behind his expected grade placement at the time 
of testing, as shown by previous years’ testing done by schools. Within 
each experimental and control school, the lowest achieving students were 
selected first, in order, up to the required number of students per 
school. In those cases where previous scores in both reading and math 
were available for a student, an average of the two scores was taken to 
establish one decrement upon which to rank the students. 

In the case of comparison students within the experimental 
schools, the next 50 lowest achieving students were selected after the 
100 experimental students had been selected* Membership in the special 
treatment groups was determined by student designation through other 
means into the special programs. 

It was not feasible to follow the basic student selection 
process described above in every case. The basic lists of students for 
selection x^ere made up during the summer preceding the school year, based 
upon already existing test data. Several changes usually occurred in 
each grade in districts because some students moved from the school. 

In &uch cases, replacements were designated locally after school had 
begun. In other cases, if previous year's test data were not available, 
teacher recommendations about low-achieving students who could 
benefit from the program were used to place students. This occurred 
frequently in Grade 1, where previous test data were usually lacking. 
Therefore, recommendations from kindergarten teachers, any readiness 
data and eligibility based on low-income status were used to place 
first-grade students. In general, supplemental means to identify 
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students for experimental and control groups were used extensively in 
some districts with high mobility rates; however, the prerdrawn list 
of students formed the basis for selection decisions in most school 
districts . 



Experimental Procedures 

Evaluation of the effectiveness of the experimental programs 
has been accomplished through standardized testing conducted at the 
beginning and end of the 1970-71 school year. One standardized test 
was administered to experimental, control and other groups in each 
grade.* Pretesting took place within 10 days of the start of school 
and posttesting commenced 15 days before the end of school. For 
posttesting, an alternate form of the same pretest instrument was used.** 

In addition, "follow-up" testing was accomplished in selected 
sites and grades within sites during the late fall of the current 
(1971-72) school year, using the same test instrument. Both previous 
experimental and control group students were tested in these sites. 

Sites and grade levels within each site where follow-up testing was 
accomplished, along with the criteria used for selection of these sites 
and grades, are given in the Results Section of the report. 

Evaluation of the experimental programs is made mainly in 
terms of experimental versus control group test achievement. However, 
additional comparisons are made with the comparison and special 
treatment groups, when achievement data are available for these groups. 
Raw score units are used as the main basis for this evaluation as 
opposed to grade equivalence units. This is because raw score units, 
while less meaningful to the casual observer, do not possess psychometric 
distortions which might affect the results of statistical analyses. 



* Comparison group posttesting was done only in selected sites and 
grades, due to the absorption of comparison group students into the 
experimental group, at several grade/sites. Sites and grades where 
comparison groups were posttested are given subsequently in the 
Results Section of the report. 

** An exception to this occurred in Grade 1, as described subsequently 
in the section on Achievement Measures. 
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For purposes of payment to technology companies, experimental 
students only were administered the reading and math portions of an 
additional standardized test, on a pre- and post-basis. Each child was 
assigned randomly to take one of three tests in use for each grade and 
subject and took an alternate form of the same instrument for a posttest. 
Payment to the contractors was based upon experimental student gains shown 
between these pre- and posttests, measured in grade equivalents. 

It is important to keep the distinction between evaluation and 
payment testing in mind. Evaluation testing is used to compare the achievement 
of experimental to control students, measured in raw score points on one set 
of instruments. The evaluation analysis is less in terms of gain for the ex- 
perimental students than it is the comparison of achievement -aval for E 
and C students at the end of the year, with the pre-test being only one con- 
t- ibutor to that status. Payment testing, on the other hand measured gain 
on a pre-post basis in terms of grade equivalents for experiner fcal students 
only as a basis for contract settlement. Battelle has not analyzed payment; 
test results, since these results do not contribute to the e"Vo_nation analysis. 

In addition to contractor payment based on gains in standardized 
tests, up to 25 percent of contract payments was based upon students' 
performance on interim tests oriented to the curricular objectives of each 
company. These tests were given at five times during the school year, in 
each grade. Although useful for payment purposes, the interim tests were 
not used for evaluation, because of the multiplicity of tests and objectives 
involved . 

Finally, for evaluation Battelle's Columbus Laboratories 
distributed a Parent Questionnaire in early 1971 to all parents of the 
students in all groups. This questionnaire contained items concerning 
the attitudes of parents toward education in general and toward special 
education programs, the educational future of their children, and 
information about the educational, occupational and financial status of the 
parents. The results of this questionnaire are summarized in subsequent 
sections of this report. In addition to the Parent Questionnaire, informa- 
tion on the race, sex, age, and school attendance (1970-71 and 1969-70) of 
experimental and control students which had been kept by local project 
personnel was made available to Battelle for the evaluation analysis. 
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The Incentives Only School Districts 

In order to provide an assessment of performance incentive 
contracting without the use of a private educational technology company, 
0E0 contracted with two school districts to sponsor "Incentives Only" 
projects. The districts were Mesa, Arizona, and Stockton, California. 
Each of the districts subcontracted with its local educational 
association to test the use of incentives in regular classrooms. 

Apart from the incentives, all other aspects of the curricula and 
classroom projects were largely unchanged. 

The local educational associations signed incentive contracts 
which specified hat the amounts to be earned were dependent upon gains 
achieved on stancardized tests used for payment. The disposition of 
funds earned, whether to reward the students or the teachers themselves, 
was left to the local educational association. In both Mesa and 
Stockton, most of the initial spending was to provide incentives to 
students, not to reward teachers. However, incentive payments to 
teachers will be made. 

The hypothesis underlying these two programs was that low- 
income, low-achieving students instructed for one year by regular 
school techniques to which the use of incentives had been added 
would register better achievement on standardized tests than a similar 
control group of students not receiving the incentives. As with the 
eighteen technology company sites, the control students were usually 
designated in different school buildings in order to prevent any 
confounding of experimental versus control comparisons by having the 
effect of the experimental programs "rub off" or become assimilated 
through adjacent classrooms. The experiment was to be conducted 
within each of Grades 1, 2, 3, and 7, 8, and 9, with approximately 
100 experimental and 100 control students in each grade. Experimental 
and control students within each grade were given standardized 
achievement tests at the beginning and end of the school year, in 
order to evaluate program impact on student achievement. 
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In the above basic matters of design, the two projects at 
Mesa and Stockton were similar to the projects in the other eighteen 
districts involved with private educational technology companies. Also, 
the standardized achievement measures used for evaluation and payment 
testing were the same as for the technology company sites, as well as 
tba same basic testing design and procedures for administering these 
tests . * 

The incentives only sites should not be considered comparable 
to the other eighteen in terms of techniques and results, however. The 
"Incentives Only" districts were selected later than the other eighteen 
districts and pretesting there took place more than one month after 
school began. Unlike the eighteen sites with performance contracts in 
which innovative hardware, software, and incentives were all being 
tasted, the Incentives Only" districts were testing only the use of 
incentives added to regular class offerings. Unlike the eighteen 
sites where 0E0 and the schools contracted for the entire educational 
package, funds were provided only for the incentives, plus administrative 
expenses., For the above reasons, results and conclusions concerning 
experimental program impact at Stockton and Mesa are given in a 
separate section of this report. The reader is referred to another 
report** on Stockton and Mesa for a description of the experimental 
programs implemented at each site, description of the target population 
involved, a summary of pre- and posttesting conditions, and interim 
results and conclusions of statistical analyses. 



* The one exception to this occurred at Mesa, where the junior high 
students did not receive a payment test, but only the evaluation 
test. 

**Interim Report on the 0E0 Experiment in Educational Performance 

Contracting: The Incentive Only Sites. February 7, 1972. Battelle, 

Columbus Laboratories, 505 King Avenue, Columbus, Ohio, 43201. 
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DESCRIPTION OF TECHNOLOGY COMPANY PROGRAMS * 

In this section, a general summary is given of the six 
technology company programs. To facilitate comparisons between the 
programs, each company program is described with respect to: 

(1) personnel and training, (2) curriculum and materials, and (3) use 
of incentives. 



Alpha Learning Systems , Incorporated 
Personnel and Training 

Alpha trained and supervised teachers that were employed by the 
public schools, along with paraprof essionals from the community, to carry 
out their program. The company worked with teachers already assigned to 
the target schools selected for the experiment, training them in instruc- 
tional and contingency management, flow charting, and the development of 
individualized programs. 

Alpha was the only company utilizing teachers employed by the 
schools; in the other five companies, the teachers were employees of the 
company. 

Curriculum and Materials 



An oral-phonics approach was used for beginning reading. The 
program combined the Miami Linguistic Series, Language Master Series, 
McGraw-Hill Sullivan Series and some special materials developed by the 
Southwestern Educational Cooperative Laboratories. Once basic reading 
proficiency was established, greater emphasis was placed on written 



* Summarized from the "Final Report to the Office of Economic Opportunity 
Performance Incentive Remedial Education Experiment”, August 31, 1971, 
Education Turnkey Systems, Inc., Washington, D.C. 20036. 
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programs— e. g. , SRA language labs end the Economy Series. The beginning 
math program consisted primarily of TMI-Gr oilier , progressing to Addison- 
Wesley. The junior high programs _oniained the same basic materials, but 
were exranded to r::. elude SRA’s heading for Understanding and Reader’s 
Digest Skill Bum ars. Alpha used a wide range of individualized, self- 
paced materials, a_. 1 of which are available commercially. Hardware was 
kept to a minimum. 

Use of Incentives 

An explicit system of contracting between the student and teacher 
for curriculum and available time was developed. Given a prepared range 
of tasks for one day, the student determined the order in which he performed 
the tasks and the reinforcements he would experience upon completion of a 
prescribed proficiency in each task. Alpha employed both intrinsic and 
tangible incentives with the students involved in the program. The former 
consisted of free time in which the student could do as he desired. A 
"free room" stocked with items the student enjoyed was made available. 
Greater levels of student learning efforts were rewarded by tangible incen- 
tives via macro contracts. These were in terms of money or additional free 
time. 



Learning Foundations , Incorporated 
Personnel and Training 

Learning Foundations hired local individuals to serve as project 
"administrators" and program "supervisors". Paraprof essionals were used 
extensively to produce a supervisor-student ratio of one to five in the 
elementary school and one to seven at the secondary level. Originally, 
the "administrator", a certified teacher, was responsible for 600 students. 
Other certified teachers were added later in one site because of political 
and legal considerations. 
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Curriculum and Materials 

Learning Found afcir relied heavily on the use of teaching 

machines. The auto-tutor, v role-speed reader, EFI Card Reader, linear 
tutor, and self-pacing aud_ rial projector comprised the hardware. 

The Craig program was used rr msively for instructional and testing pur- 
poses. The Sullivan program ■ reading and math were also used con- 
siderably in the Learning F r ..cions program. The other materials were 
primarily learner paced and ranad to individualization of instruction. 
All the hardware is commerc; ~ available. 



Use of Incentives 



There was a heavy emphasis on incentive in the Learning Foundations 
program. Students were awardee points for achievement in areas of attendance, 
speed, and comprehension. These points were in the form of play money which 
could be used to "purchase" listed gift items. Some of the rewards further 
reinforced the learning process--e.g . , dictionaries, telescopes, etc. In 
addition, bonus points were given for greater learning and attendance effort. 

Plan Education Centers, Incorporated 
Personnel and Training 

Plan hired equal nuzibers of professionals and paraprofessionals 
locally for their learning centers. The Project Administrator was an 
individual hired specifically for this project. Preschool workshops 
and inservice workshops were held to train the staff on site. 



Curriculum and Materials 



Plan’s learning centers were stocked with a variety of materials, 
the largest of any of the systems proposed. Use of hardware was minimal. 

In the Plan program, ma theme • vas taught as a language, as a statement- 





17 



generating system. By combining the teaching of reading and mathematics, 
especially at the early stages, and by interrelating vocabulary, sentence 
structure and comprehension, Plan coordinated learning in both areas. The 
Sullivan reading and mathematics materials along with SRA materials 
constituted the main components of the total program. The materials were 
largely learner paced and geared to individualization of instruction. 

There was some small group instruction, however, as Plan’s program called 
for grouping by disability for intensive instruction for short periods. 
This was primarily tutoring by the paraprofessionals. 

Use of Incentives 



Plan did not provide teachers and students "tangible" rewards. 
Rather, through careful diagnosis and prescription, the student would be 
rewarded intrinsically as he progressed through the material. The extent 
to which rewards for students were provided by teachers did not differ 
significantly from those provided by other teachers in the system (e.g., 
verbal reinforcement, smiles, etc.) 

Quality Educational Development, Incorporated 

Personnel and Training 

Local personnel were used as teachers and aides in the Q.E.D. 
program wherever possible. Individuals, not originally employed by Q.E.D. , 
were hired specifically for this project as Project Administrators. 

An equal number of paraprofessionals and professionals were hired and used. 

Curriculum and Materials 



Two programs were used to teach beginning reading — at two sites, 
the Furguson ITA Program was used initially although later replaced; at 
the other, the Evans Reading Program. The Sullivan Reading Program was 
next in the progression. At the junior high school level, the Job Corps 
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Grading Reading Selections were added. In mathematics for both elementary 
and secondary levels, Houghton-Mif f lin T s Modern School Mathematics; 
Structure and Use Program was used in conjunction with the Stockton 
Individualized Learning System. 

Q.E.D. based their instructional systems on programs of proven 
effectiveness although not widely used. This approach made optimum use 
of inexpensive hardware for the presentation of materials and provided a 
high teacher-machine to student ratio. A program of individually pre- 
scribed instruction was used consisting of a placement system, pre- and 
post-tests, and individualized materials enabling the child to progress 
at his own rate. As gaps appeared in the existing materials system, Q.E.D. 
created individualized, self-paced materials designed to modify or supple- 
ment the curriculum. 

Use of Incentives 

Contingency management as a direct application of reinforcement 
technology was utilized by Q.E.D. in the following way. The students were 
issued "credit cards" enabling them to receive points to their credit as 
they achieved various modes of success. They could then acquire articles 
from a "catalogue store" or "buy" time to pursue a preferred activity. 

Rolls of tickets as one might find at an amusement park were also used 
for this purpose. 



Singer/ Graf lex 



Personnel and Training 

Local professionals and paraprof essionals were employed at a 
ratio of approximately five to two. Leadership and supervisory positions 
were filled by Singer /Graf lex veterans with the majority of the project 
staff hired locally. An intensive training program for all staff at the 
beginning of the project was considered essential,, and periodic training 
sessions were conducted throughout the term of the contract. 
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Curriculum and Materials 



Singer relied heavily on the Job Corps Reading and Math Programs. 

The Sullivan Reading Program, SRA materials, and the Sullivan Math Program 
received special emphasis. In addition, the Singer Right to Read Program 
was used. Some audio-visual materials were utilized. These included 
filmstrips, tapes, controlled readers and language masters. Singer used 
commercially available materials in whole or in part depending upon the 
needs of the student. They were largely learner paced, based on individualized 
instruction. 

Use of Incentives 

The Singer approach called for intensive use of incentives. 
Achievement certificates and emblems were awarded. Students were encouraged 
to accumulate achievement points and bonus achievement points to exchange 
for catalogue merchandise. At the secondary level, achievement was reward- 
ed with free time in which the student engaged in an activity of his choice. 

Initially, social behavior was stressed and rewarded, e.g., 
proper use of the materials, cooperation with the instructors, etc. The 
emphasis then shifted to the reinforcement of appropriate learning behavior — 
the satisfactory completion of an assigned lesson. In the beginning, 
tangible items for all the students were used extensively. Later in the 
project, especially at the secondary level., intangible rewards such as 
free time to play games , movies , a trip for free hamburgers and milkshakes 
were stressed. 



Westinghouse Learning Centers 
Personnel and Training 




The program staff was composed of a combination of certified 
teachers and paraprof essional aides. In addition, a Center Manager was 
employed at each location to serve as the overall supervisor. The center 
staff and aides were recruited from the lceal community. The professionals 
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in some cases, were former school teachers, and the aides were chosen on 
the basis of their potential for establishing rapport with the student 
population. All staff members were trained in the operation of the instruc- 
tional system by Westinghouse personnel. Periodic training was also con- 
ducted throughout the duration of the project. 

Curriculum and Materials 

" — * ■* - • ■ — -■ ■ - — 7 — 

To teach beginning reading, Westinghouse relied primarily on 
the BEL Reading Readiness Program. The Sullivan Reading Program followed 
and was used in conjunction with SRA labs ^nd Charles Merrill materials 
as the main components of the reading program. For arithmetic, the 
Sullivan Basal Mathematics Program was heavily relied upon, supplemented 
by workbooks and audio-visual instructional materials when needed. 

Westinghouse’s instructional sequence intended to teach specific 
skills, using portions of commercially available materials. The curriculum 
included materials that were designed to be self-instructional, learner- 
paced, and those that taught the skills outlined in the curriculum. Also, 
the use of instructional hardware was observed in the Westinghouse system. 
For the most part, standard cassette tape players with headsets and reel- 
to-reel tape players were required for use with certain materials. 

Use of Incentives 



The contingency management motivational method employed in the 
operation of the Westinghouse program was characterized by contingency 
contracts — a daily agreement between the staff and the student stating 
precisely how many prescribed instructional sequences the student would 
complete in order to earn some free time. Implementation of this design 
involved the designation of two areas in the classroom — one area being a 
study area; the other, an activities area. 
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Tabula r Summary 

Some aspects of the experimental programs are summarized in 
tabular form in Table 1 on the following page. The table shows, for 
each technology company, percent of paraprof essionals employed, 
student-staff ratios, and type of incentives and instruction used. 
Both teachers and paraprof essionals are included in the student-staff 
ratio calculations. 
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TABLE 1. COMPARISON OF PARTICULAR ASPECTS OF EXPERIMENTAL PROGRAMS 
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DESCRIPTION OF THE TARGET POPULATION 



The major purpose of this section of the report is to present 
descriptive information concerning some of the more important variables that 
characterize the socioeconomic status, social and home background, and 
achievement levels of students who participated in the experiment . Of 
particular concern are family income and achievement levels of students, 
since the experiment was directed toward "low achieving" students from 
"low income" families. However, descriptive data are also presented on 
the racial composition of students in the study, along with the educational 
and occupational status of the students' families, in order to afford the 
reader a more complete description of the type of student for which the 
results and conclusions of this study apply. 

In this section of the report, achievement level data are taken 
from the pretest administration of the evaluation test, to characterize 
the entry level achievement of students participating in the study. The 
racial composition of students was obtained from student locator cards, 
a system of student characteristics maintained by the Project Directors 
at each site. Data on family income, and educational and occupational 
status of students' parents, were secured from a family background 
survey, consisting of a questionnaire sent to students ' parents (de- 
scribed below) o 

Data are presented separately for experimental and control 
groups, so as to show differences between the two groups. The imple- 
cations of these group differences for the analysis, along with the 
methodology for taking into account group differences, is treated 
subsequently in the analysis sections of the report. 

Data are presented for students who were in the experimental 
or control group for the entire school year*. These full-year students 



* Table A-l, Appendix A, gives the number of full year students, by 
site, group, and grade. 
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constitute the basis for the analysis. Sufficient achievement test data 
were not available for separate analyses of "dropouts", or students 
entering the experimental group program after its onset. However, 
descriptive data characterizing the initial entry achievement level of 
dropouts is given at the end of this section, to serve as a basis 
for comparing students who remained in the program the entire year 
with those who dropped out. 

Securing Family Background Information 

A family background survey was implemented during the month 
of December, 1970 s for purposes of gathering parent attitudinal and 
socioeconomic data. A total of 27,018 participant identified question” 
naires were delivered to each of the twenty project directors for program 
participants identified in Baftelle's master file. An additional 100 
questionnaires were sent to each site for student replacements to the 0E0 
experiment who were not identified previously for Battelle in the original 
master list. Additionally, a Spanish version of the questionnaire was 
prepared by Battelle and sent to those sites requesting dual forms: 

Grand Rapids, Stockton, and Bronx. 

The method of delivering questionnaires to parents was as 
follows. First, site Project Directors distributed the questionnaires 
to teachers of the students in the study. The students were then in- 
structed to take the questionnaires home to their parents. Parents then 
filled out the questionnaire in accordance with enclosed instructions, and 
then returned the questionnaire directly to Battelle. As part of the 
instructions, parents were assured that their answers to the questions 
would be treated in a confidential manner, with only concerned Battelle 
staff members seeing their responses. 

Two methods of follow-up were used. First, four days after the 
questionnaires, were distributed to students, each student received a 
reminder card, which they were instructed to take home to their parents. 
The card "reminded" parents of the significance of the study, and again 
asked for their cooperation in responding to and returning the question- 
naire to Battelle. 
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The second method of follow-up consisted of sending out a 
second wave of questionnaires to identified non-respondents, again using 
the ‘'student delivery" system. This second wave was sent out approxi- 
mately three weeks after the first questionnaire was to have been returned 
to Battelle. 

Table B-l (Appendix B) presents data relative to the return 
rate of parents, including all follow-up efforts. Data are presented 
for each site, and for each group and grade within each site. Return 
rates are calculated on the basis of the student population in the 
experiment at the time questionnaires were received at the school 
district . 

Inspection of these data show variability of return rates be- 
tween sites, grades, and experimental versus control groups. However, 
return rates on the order of 40-50 percent are not atypical, and 
represent a rate usually considered normal for the method of survey 
implementation used in this study. Nevertheless, this leaves missing 
data for at least 1/2 of the students' families. Return rates are 
discussed below, where rates are given for full- year students on the 
particular variables of family income, education, and occupation. 

Appendix G presents a copy of the Parent Questionnaire sent 
to students 1 families. Items 32, 22, and 28 in the questionnaire were 
the basis for the income, educational, and occupational data. 

Characteristics of Participants 
Race, Education, Income, and Occupation 

As stated previously in the "Description of Experiment" section, 
many different types of school districts were selected to be in the 
experiment, geographically scattered across the United States. This 
diversity is reflected in variations of student characteristics from 
site to site, expecially with regard to race of the student, and family 
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income. For example, for race, there may be one, two, or three pre- 
dominant races represented by students at a particular site, and the 
race(s) which predominate vary from site to site. Consequently, for 
race and income, descriptive information is given on a site by sir 
basis only. 

Educational and occupational characteristics of students 1 
families also varied from site to site, but to a lesser degree than 
for income and race. Consequently, both site by site as well as data 
aggregated across <?ites are presented for these variables. 

Race of Students . Table D-l in Appendix D shows the per- 
centage distribution of race, for each technology company site, and for 
experimental and control groups within a site. For each site, data 
given are for Grades 1, 2, 3, 7, 8, and 9 combined, for full -year 
students. Also given is the sample size on which these race data are 
based, i.e., the number of full-year students for which data are 
available. Finally, a “response rate" is given, which is the percent- 
age of full-year students for which race data were available. 

Response rates for the race data are generally high, most 
frequently being in the high 90' s, as inspection of Table D-l shows. 
However, for the control groups at four sites (Hartford, Philadelphia, 
Rockland, and Taft), race data were not available from the sites. 

Inspection of these data show differences between experimental 
and control students in racial composition within sites. Thus, of 
the 14 sites for which race data were available for both experimental 
and control groups, for 10 of these sites the percentage of Whites in 
the control group is higher than the percentage of Whites in the experi- 
mental group. The differences in these percentages is often quite 
marked. For example, at Anchorage, 91 percent of the control group is 
White, whereas only 54 percent of the experimental group is White. As 
another example, at Hammond, the percentage of Whites in the control and 
experimental groups ,is 87 and 57 percent respectively. 
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Inspection of the data in Table D-l also reveal appreciable 
variability in racial composition of the stuaent population from site 
to site. This is illustrated in Table 2, which lists the predominant 
races for each site, for the experimental and control groups. "Pre- 
dominant” races are defined by listing those races (or that race) that 
accounts for at least 85 percent of the students, by taking the most 
frequently occurring race, the next most frequently occurring race, 
etc. , until at least 85 percent of the students are accounted for. 

Within the experimental and control groups, races are listed (from left 
to right) in the order of decreasing dominance. 

Inspection of Table 2 shows that in eight sites, there was 
only one predominant race. In four of these eight sites (Dallas, Hartford, 
Jacksonville, and Philadelphia), the Black race predominated; In three 
other sites (Portland, Rockland, and Selmer) , the White race predominated; 
in one site (Taft), a predominance of Mexican-Americans is shown. 

In seven of the sites, Blacks and Whites predominated. These 
seven sites were Athens, Grand Rapids, Hammond, Las Vegas, McComb, Seattle, 
and Wichita. In Anchorage, Whites, Eskimos, and Blacks accounted for at 
least 85 percent of the students in the study; at Fresno, the population 
consisted mainly of Mexican-Americans and Whites; and at Bronx, the 
predominant races were Puerto Ricans and Blacks. 

In summary, often marked differ ences appeared between experi- 
mental and control groups within a site in racial composition. Further, 
across sites, a diversity of racial composition of both minority and 
majority group membership was represented in this experiment, reflecting 
the diversity of school districts selected t r * e in the study. 

Family Income . Table 3 shows the median total family income 
for parents of the experimental and control students, for each of the 
eighteen sites in Grades 1, 2, 3, 7, 8, and 9, combined. Also shown is 
the percentage of students' families having an income less than the 
national average, by site and group (E or C) . 
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TABLE 2 . PREDOMINANT 


RACES OF FULL-YEAR STUDENTS, 


BY SITE AND GROUP 


•Site 


Experimental Group 


Control Group 


Anchorage 


White, Eskimo, Black 


White 


Athens 


Black, White 


White, Black 


Bronx 


Pureto Rican, Black 


Puerto Rican, Black 


Dallas 


Black 


Black 


Fresno 


Msx. Amer., White 


Mex. Amer., White 


Grand Rapids 


White, Black 


White, Black 


Hammond 


White, Black 


White 


Hartford 


Black 




Jacksonville 


Black 


Black 


Las Vegas 


Black, White 


White, Black 


McComb 


Black 


Black, White 


Philadelphia 


Black 


— 


Portland 


White 


White 


Rockland 


White 




Seattle 


White, Black 


White 


Selmer 


White 


White 


Taft 


Mex. Amer, 


— 


Wichita 

0 


Black, White 


White, Black 
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TABLE 3 . MEDIAN TOTAL FAMILY INCOME AND PERCENTAGE OF FAMILIES HAVING 
AN INCOME LESS THAN THE NATIONAL AVERAGE FOR FULL-YEAR 
STUDENTS, BY SITE AND GROUP 
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Median incomes shown can be compared to the overall national 
median income of $9,867*, as of March, 1971. All median incomes are 
less than this nationwide median, with the exception of Anchorage, which 
shows a median of $10,565 for the experimental group, and $14,896 for 
the controls.** 

However, median incomes vary widely from site to site, with 
lows occurring at McComb, Dallas, and Taft (Median incomes are on the 
order of $2,000 to $3,000 at these three sites). Other sites show median 
incomes varying between $4,000 and $8,000, generally. 

The percentage of families having an income less than the 
national average also varies from site to site, although these percentages 
are generally high. Thus, of the 36 site/group-, combinations , 24 of the 
combinations show at least 75 percent of the families having an income 
less than the national average. 

All the above cited descriptive data needs to be interpreted 
with some caution, in view of the response rates shown in the table. 

Response rates from Bronx, Hartford, Las Vegas, Philadelphia, and Taft 
are particularly low. For example, at Taft, in the control group, income 
data were available on only 18 percent of the full-year students. 

The data in Table 3 reveal E/C group differences in income. For 
example, in Anchorage, the median income for the control group is on the 
order of $4,000 higher than for the experimental group. However, the 
magnitude and direction of group differences in income within a site is 
best illustrated in Appendix E, which gives median incomes by grade level, 
as well as by site and group. For example, at Jacksonville in Grades 1 j 
2, and 3, the experiments 1 group has appreciably highei median incomes than 
the control group, whereas the reverse is true for the secondary level (i.e., 
the control group exhibits higher median incomes than the experimental group 
in Grades 7, 8, and 9). Differences such as these tend tc be obscured in 
Table 3, both in terms of magnitude, and direction of group differences. 



* Taken from "Current Population Reports: Consumer Income", U.S. Depart- 

ment of Commerce, Bureau of the Census. Series P-60, No. 78, May 20, 1971. 

** Higher median incomes are to be expected in Anchorage, since the cost of 
living there is substantially higher than in the 48 States. 
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From an analysis point of view, the data given in Appendix E by grade 
are more descriptive, since, as described subsequently, the major unit 
of analysis is at a grade/site level. As another example, at McComb, 
families of elementary students in the control group have appreciably 
higher incomes than families of experimental group students, with much 
smaller differences between groups at the secondary level. 

Father * s Education . The percentage distribution of father's 
education for each site is shown in Appendix F, in three categories: 
completed less than high school; completed high school; completed more 
than high school. The latter category is an aggregate one, and includes 
some or completion of college, and/or some or completion of vocational 
school after high school, and/or some or completion of business school 
after high school. 

Examination of these data shown certain E/C group differences 
within sites. Thus, in 9 of the 18 sites, the control group has a 
smaller percentage of fathers with "less than high school" than does 
the experimental group, and a higher percentage of "more than high 
school" than does the experimental group. For example, at Athens, 51 
percent of the control group students have fathers with less than a high 
school education, whereas 64 percent of the experimental group falls in 
this category; 22 percent of the control group students at Athens have 
fathers with more than a high school education, compared with 10 percent 
in the experimental group. 

A salient feature of the education data in Appendix F is the 
relatively large percentage cf fathers having less than a high school 
education in both experimental and control groups. In general, this 
percentage is approximately 50, in each site. This is reflected in Table 4 
on the following page, which shows that 50 percent of the fathers of 
students in the experimental group and 47 percent in the control group 
have less than a high school education, for all sites combined. Relatively 
few fathers have more than a high school education (22 percent in the 
experimental group, and 24 percent in the control- group). 
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TABLE 4. PERCENTAGE DISTRIBUTION OF FATHER'S EDUCATION 

FOR FULL- YEAR STUDENTS, BY GROUP, FOR ALL SITES 



Educational 

Level 


| Experi- 
mental 
Group 


Control 3 
Group I 


© Less than 
High School 


50 


47 


© Completed 1 

High School ? 


28 


29 


© More than • 

High School 


22 


24 I 




100 


100 I 


Sample Size: 


3917 


3332 


Response Rate: 


44 


39 



TABLE 5. PERCENTAGE DISTRIBUTION OF FATHER'S OCCUPATION 
FOR FULL -YEAR STUDENTS, BY GROUP, FOR ALL SITES 



Occupational 

Category 


mental 

Group 


Control 

Group 


© Semi or 
Unskilled 


47 


43 


© Skilled 
Manual 


28 


30 1 


© White Collar | 
Professional 
or Business 


I 25 


27 




100 


100 


. _h 1 








Sample Size: 2711 

Response Rate: 31 



2428 

28 
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Father's Occupation . The percentage distribution of father's 
occupation is shown in Appendix G, for each site and group. Three 
categories are given: (1) semi or unskilled, (2) skilled manual, and 
(3) white collar, professional, or business. Semi or unskilled includes 
occupations such as cafeteria or laundry work., unskilled factory work, 
farmer's helper, factory machine operator, gas station worker, and foundry 
worker. Skilled manual work includes the skilled trades, such as welder, 
electrician, and applicance repairman. White collar, business or professional 
includes clerical or sales workers, small and large business owners or 
managers, and high and low paid professional work. 

As summarised in Table 5, the preponderance of students' fathers 
across sites have occupations falling in the semi or unskilled category. 
Percentages are 47 and 43 percent, for the experimental and control groups, 
respectively. Approximately 30 percent of the fathers for the E and C 
groups fall in the skilled manual category, and approximately 25 percent 
fall in the white collar, business, or professional category. 

As with the variables of educat ‘ n, income, and race, E/C group 
differences within sites occur. For example, at Selmer, the occupational 
"level" is higher in the experimental group than the control group. Thus 
60 percent of the control student's fathers fall in the "semi or unskilled" 
category, whereas only 45 percent of the experimental student's fathers have 
occupations in this category, with a greater percentage of fathers in 
the white collar, professional, or business category. 

The response rates for occupational data are somewhat lower than 
for income and educational data, due to the absence of some fathers from 
the home, and because some fathers were not currently employed. Thus, the 
descriptive data in Appendix G need to be qualified accordingly „ 




Achievement Levels 

Tables H-l, H-2, H-3, and H-4 in Appendix H present data 
relative to the entry level of students in the experiment. 

Tables H-l and H-2 in Appendix H give mean grade equivalency 
(GEQ) values for reading (Table H-l) and for math (Table H-2), by site, 
group, and grade. For Grades 2, 3, 7, 8, and 9, the tables give the grade 
equivalency corresponding to the mean raw score on the evaluation 

x 6 
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pretest, for full-year students. For Grade 1, the tables given the sLanine 
value* corresponding to the mean raw score on the evaluation pretest, again 
for full-year students. 

Inspection of Tables H-l and H-2 show that "average" grade 
equivalency values are almost universally below grade level, across sites 
and E and C groups, with the amount below grade level (the grade level 
"decrement") generally increasing with grade level. However, although 
students are generally below grade level in both experimental and control 
groups, differences between E and C groups in entry level are nonetheless 
apparent, as examination of Tables H-l and H-2 show. In this connection, 
at the secondary level, average grade equivalencies are generally higher 
for the control groups. Thus, of the 106** site/grade/subject area 
combinations at the secondary level, the control group has a higher grade 
equivalency entry level in 84, or approximately 80 percent of these com- 
binations. At the elementary level, although group differences often 
exist, the tendency for the control group to have higher entry levels 
than the experimental group is much less marked (the control group has 
a higher entry level in 66 of the 106 possible grade/site/subject area 
combinations at the elementary level). 

Tables H-3 and H-4 in Appendix H give percentage of students 
below grade level, for Grades 2, 3, 7, 8, and 9, for each of reading and 
math. For example, for Anchorage, E group, Grade 8 ia reading, 91 percent 
of the students registered below grade level on the evaluation pretest. 
Inspection of these tables show very large percentages of students below 
, grade level. There are, however, some very few exceptions to this in 
certain sites and grade/groups within sites, as inspection of Tables H-3 
and H-4 show. 



* A stanine is a transformation of a raw score, ranging from 1 to 9, 
such that the lowest and highest 4 percent of the raw scores are 
assigned the values 1 and 9, respectively; the next lowest and high- 
est 7 percent are 2 and 8; the next lowest and highest 12 percent 
are 3 and 7; the next lowest and highest 17 percent are 4 and 6; 
and the middle 20 percent are 5. 

** (18 sites) (3 grades) (2 subject areas) = 108 grade/site/subject 
combinations at the secondary level, less 2 combinations ip Grade 9 
in Bronx, where data were not available. 
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The very large percentages of students below grade level are 
summarized in Table 6 below. This table shows percentage of full-year 
students below grade level by grade, subject area, and group, for all 18 
sites combined. Percentages below grade level are generally in the mid 
80 ' s to high 90's. The smallest percentages are 83 (Grade 9, reading; 
Grade 2, mathematics) and 84 (Grade 2, reading). 



TABLE 6. PERCENTAGE OF FULL-YEAR STUDENTS BELOW GRADE LEVEL 
FOR READING AND MATHEMATICS, BY GROUP AND 
GRADE, FOR ALL SITES 





Reading 


Mathematics 


Grade 


Experimental 


Control 


Experimental 


Control 


2 


91 


84 


85 


83 


*3 


97 


86 


95 


86 


7 


96 


88 


96 


91 


8 


95 


91 


96 


92 


9 


93 


83 


97 


90 



The entry level achievement of students in the experiment 
is also summarized in Table 7, in grade equivalence units, for all 
eighteen sites combined. For Grades 2, 3, 7, 8, and 9, the table gives 
the grade equivalence corresponding to the mean raw score for the eighteen 
sites on the evaluation pretest, for reading and mathematics and for 
experimental and control groups. Stanine values are given for rade 1, 

The data in Table 7 are portrayed graphically in Figure 1, 
where mean pretest grade equivalencies (GEQ's) are plotted for each grade 
level, group, and subject area (reading or mathematics). Grade 1 is 
not plotted, since data are not available in grade equivalence units 
for the Grade 1 evaluation pretest. 

As shown in Figure 1, both experimental and control students 
in Grade 2 start out at about the same amount below grade level (as 
indicated by the vertical dista T ce from the 45 degree line), and both 
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FIGURE 1 

o 

ERIC 




Grade Level 




Grade Level 



. OVERALL AVERAGE PRETEST GRADE EQUIVALENTS FOR 
EXPERIMENTAL AND CONTROL GROUPS, FOR EACH GRADE 
IN READING AND MATHEMATICS. 
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E and C groups get further behind grade level with each succeeding year 
in school. Also, as the grade level increases, the separation between 
experimental and control groups increases, and more so for reading than 
for mathematics. For each group in each subject area, the rate at which 
students fall behind grade level seems to be about constant from year to 
year, as indicated by the fact that each of the four plots is essentially 
a straight line. Finally, both E and C students fall further behind 
grade level in reading than in mathematics, as shown by the lower "growth 
rates" (or smaller slopes of the lines) for E and C groups in reading than 
in mathematics. 



TABLE 7. OVERALL MEAN PRETEST GRADE EQUIVALENT VALUES FOR READING 
AND MATHEMATICS, BY GROUP AND GRADE, FOR ALL SITES 





Reading 




Mathematics 






Experimental 


Control 


Experimenta 1 


Control 


Grade 1 


2* 


3* 


2* 


3* 


Grade 2 


1.5 


1.6 


1.4 


1*4 


Grade 3 


2.1 


2.3 


2.2 


2.3 


Grade 7 


4.5 


5.0 


4.7 


5.1 


Grade 8 


4.8 


5.6 


5.4 


5.9 


Grade 9 


5.6 


6.4 


6.0 


6.6 



* Stanine values for Grade 1 



Other Variables 



In addition to the variables of race, income, education, occu- 
pation, and pretest, experimental and control groups were examined with 
respect to several other variables, many of which were attitudinal 
items from the Parent Questionnaire. These other variables were examined 
largely from the point of view of comparability of E and C groups, and 
thus the focus was on examining how the two groups differed on these 
variables. 
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Additional items from the Parent Questionnaire that were examined 
in order to study group differences are as follows: 

© Importance of schooling (Item 2) 

© Approval of new teaching method for own children (Item 10) 

© Assistance in doing school work (Item 18) 

© Encouragement to do well in school (Item 19) 

© Amount of schooling desired for child (item 20) 

© Amount of schooling expected for child (Item 21) 
o Employment status of husband and presence/absence 
of husband in household (Item 26) 

© Per capita income (derived from Items 30 and 32) 
o Welfare experience (Item 34). 

The Parent Questionnaire, with the full statement of each of the above 
items, is presented in Appendix C. 

In addition to the above items, data were obtained from student 
locator cards for student age, and for student attendance during the 
1970-71 school year. Experimental versus control group differences were 
also examined with respect to these two variables. 

Group differences on all of the above variables were studied in 
the following manner. For any given variable at a given grade/site, an 
examination was made of the variable's distribution for the E and C groups, 
for Grades 1, 2, 3, 7, 8, and 9 combined. If the E group's relative 
frequency distribution differed by more than 10 percentage points from 
at least one corresponding category in the C group's distribution, the 
groups were judged to differ on the variable. For example, at Dallas for 
Item 34 (welfare experience), 67 percent of the control group's families 
reported they were currently receiving financial aid, whereas only 46 percent 
of the experimental group reported that they were currently receiving aid. 
Thus, at Dallas, the E and C groups were judged to differ on this item. 

The results of this analysis are shown in Table 8. An "X" 
appears wherever an E/C difference exists according to the criterion 
described in the previous paragraph. 
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The table also contains the variables of race, total family 
income, father's education, and father's occupation, as well as the 
variables mentioned immediately above, in order to provide a summary of 
group differences for all variables. 

Of the attitudinal items analyzed from the Parent Questionnaire 
(Items 3, 10, 18, 19, 20, and 21) , of particular interest is the number of 
sites on which E and C groups differed on Item 10, where parents were 
asked how much they approved (or disapproved) of their children being 
taught by a new teaching method. Parents responded in four categories 
ranging from "approve very much" to "disapprove very much". 

As shown in Table 8, of the 17 sites where data were available, 
an E/C group difference on Item 10 was judged to exist in 13 of the 17. 
Further, in several sites, the families of the experimental group had a 
more favorable attitude toward their children being taught by a new 
method than did families of the control group. This is illustrated in 
Appendix I, which shows, for each of the 18 sites, the frequency distribution 
of responses to each of the four categories in Item 10. As can be seen there, 
in several sites, the category "approve very much" is responded to more 
frequently in the experimental group than the control group. 

As mentioned previously, the method for taking into account 
group differences as described above in assessing program impact on posttest 
performance is presented subsequently in the analysis section of the. report. 

Comparison of Full -Year 
Students and Dropouts 

In any study where experimental group members are subject to 
attrition, it is of interest to inquire whether subjects who "dropout" of 
the experimental treatment are systematically different from those who 
remain for the entire treatment. This of particular concern in this 
experiment, since, as mentioned earlier, the analysis is confined to those 
students who remained in the program the entire year. 

In this study, most students are not old enough to drop out of 
school. Thus, the reason for a student dropping out of the program can 
generally be attributed to other factors, such as the student's parents 
moving from the school district. 
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TABLE 8. ADDITIONAL VARIABLES, OTHER THAN PRETEST, ON WHICH THE EXPERIMENTAL AND CONTROL GROUPS 
WERE JUDGED TO DIFFER FOR EACH SITE 





The number in parentheses is the actual item number in the Parent Questionnaire. 

X = E/C difference greater than 10 percentage points in at least one of the categories of the variable, 
Ku « No Data 
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In order to compare full year experimental students with experi- 
mental students who dropped out of the program at seme point during the 
school year, the pretest scores of both groups were examined. In particular, 
the mean pretest scores of full-year students was compared to the mean pre- 
test scores of dropouts, aggregated across the 18 sites. This was done for 
each grade and for each of reading and mathematics within a grade. 

Results of this co* on are shown in Table 9 below. Raw score 

means are given for full-yeai. students and dropouts, along with associated 
grade equivalence values. Stanine values are given for Grade 1. 



TABLE 9 . MEAN PRETEST VALUES (AND ASSOCIATED GRADE EQUIVALENTS) FC I 
FULL-YEAR EXPERIMENTAL STUDENTS AND DROPOUTS, BY GRADE 
AND SUBJECT AREA 





Gra 


de 1 


Gra 


de 2 


! 

Grade 3 


Grade 7 


Grade 8 


Grade 9 




Read 


Math 


Read 


Math 


Read 


Math 


Read 


Math 


Read 


Math 


Read 


Math 


Full -Year 


68 


68 


32 


28 


33 


44 


40 


43 


32 


39 


38 


46 


Students 


(2)* 


(2)* 


a. 5) 


(1.4) 


(2.1) 


(2.2) 


(4.5) 


(4.7) 


(4.8) 


(5.4) 


(5.6) 


(6.0) 


Dropouts 


68 


68 


30 


28 


33 


46 


40 


42 


32 


39 


37 


44 




(?)* 


(2)* 


(1.4) 


(1.4) 


(2.1) 


(2.2) 


(4.5) 


(4.6) 


(4.8) 


(5.4) 


(5.5) 


(5.9) 


Number of 


1047 


1042 


1242 


1152 


1317 


1242 


1211 


1197 


1151 


1065 


1093 


1102 


Full -Year 


























Students 


























Number of 


196 


196 


208 


190 


185 


174 


179 


186 


216 


214 


224 


222 


Dropouts 



























*Stanine values. 



Inspection of Table 9 shows remarkably similar pretest means for 
full year students and dropouts. In terms of grade equivalencies, full- 
year students and dropouts never differ by more than one -tenth of a year, 
and the grade equivalencies (or stanines for Grade 1) are identical for 
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the two groups for eight of the twelve grade-subject combinations. In 
terms of raw score means, the full-year students and dropouts never 
aiffer by more than two raw score units, and the mean raw score values 
are again identical for the two groups for eight of the twelve grade- 
subject combinations. 

Thus, it is concluded that the overall entry level achieve- 
ment of full-year students and dropouts is the same, and that the 
dropouts are not a different group than full-year students with respect 
to achievement level. 
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ACHIEVEMENT MEASURES* 



Battelle, in its role as Test and Analysis Contractor, had 
responsibility for advising the Office of Economic Opportunity in the 
selection of standardized achievement tests for administration to students 
involved in the experimental program. In this section of the report, a 
summary is given of the criteria used for the selection of the evalua- 
tion tests, along with an identification and brief description of the 
evaluation tests selected. Identification and descriptions of the 
certification tests are given in Appendix J of this report. 

Selection of Evaluation Tests 



The criteria for the selection of the tests included the 

following: 

o The norms for the tests should be based on a rela- 
tively recent national standardization sample having 
a reasonably large number of students from large 
metropolitan and rural school district poverty areas 
as well as those from the more "average", middle class 
school districts. 

® A relatively current revision of test content that 
included subtests which sample a wide range of skills 
and knowledges of high relevance to the generally 
accepted goals and objectives of the curriculum. 

• A high degree of reliability. 

© Clear and simple directions for administering the 
test. 

® The availability of alternative forms. 

In addition to the above, it was decided that the tests 
should (1) to the extent possible, consist of a single battery of tests 



’'Abstracted from the "Summary Report on Description of Test Selection 
Rationale to Office of Economic Opportunity", January 12, 1971, Battelle's 
Columbus Laboratories, Columbus, Ohio 43201. 





44 



having several levels appropriate for students enrolled in the six grades 
and (2) reflect the generalized outcomes of the overall curriculum design 
for these grades. 

The Battelle staff met with a representative of the Office of 
Economic Opportunity to select from a listing of various achievement test 
batteries, those tests, needed to meet the requirements of the program. The 
Office of Economic Opportunity representative had researched existing 
standardized tests throughout the surraner months and presented his 

recommendations , as one input to the selection process. Each of the 
tests was reviewed * with respect to the criteria previously described. 

For this- review, information was obtained from the following sources : 

(1) knowledge gained by the Battelle staff through direct use of the tests 
in other testing programs (2) technical manuals and test construction in- 
formation (3) Buros 1 "Sixth Mental Measurements Yearbook", and (4) a report 
prepared by UCLA's Center for the Study of Evaluation, entitled "Elementary 
School Evaluation Kit".* 

In order to assure that most students would achieve some degree of 
successful performance on a test, and to reduce the dimension of test dif- 
ficulty for potential low-achieving students, it was decided, for a given 
grade, to use a level of a test that had an intended grade range that 
included the previous grade. For example, instead of seeking a test for 
ninth grade students that was designed to be given at Grades 9 through 12, 
a test was sought having an intended grade range of, say, 7 througn lO, or 
8 through 11, 

Table 10 presents a listing of the evaluation tests selected for 
each grade, along with the intended grade ranges. Two tests were selected 
for the first grade. For Grades 2, 3, 7, 8, and 9, the Metropolitan 
Achievement Test Series (1970) was selected, with the test levels used for 
each grade as indicated in the table. For Grades 8 and 9, the same test 
level (Advanced) was used in each grade, 

>■ — — — — i — l « 

*ELEMENTARY SCHOOL EVALUATION KIT, prepared by the Staff of the Elementary 
School Evaluation Project, Center for Study of Evaluation, UCLA, Develop- 
mental Copyright, 1970, 
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TABLE 10, LIST OF TESTS FOR EACH GRADE 



Grade 


Title of Tests 


Intended Grades 


1 


Stanford Early Achievement-Pretest 

California Achievement Test-Posttest 
1970 Edition (Level 1) 


Kindergarten 
to 1.5 

1,5 - 2,0 


2 


Metropolitan Achievement Test 

1970 Edition (Primary I) (Forms F and G)* 


1,5 - 2.4 


3 


Metropolitan Achievement Test 

1970 Edition (Primary II) (Forms F and G)* 


2.5 - 3,4 


7 


Metropolitan Achievement Test 

1970 Edition (Intermediate) (Forms F and G)* 


5.0 - 6,9 


a 


Metropolitan Achievement Test 

1970 Edition (Advanced) (Forms F and G)* 

i ■ . - 


7,0 - 9.5 


9 


Metropolitan Achievement Test 

1970 Edition (Advanced) (Forms F and G)* 


7,0 - 9,5 



*For each of Grades 2, 3, 7, 8, and 9, Form F was used as the pretest, and 
alternate Form G as the posttest. 
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Descriptions of Evaluation Tests Selected 



Grade 1 



Because of the scarcity of standardised achievement tests having 
a good balance of both readiness and reading items , and having norms on the 
performances of kindergarten and entering first grade children, it was 
possible to consider only two tests for measuring first grade entry-level 
achievement; The California Achievement Test, Level I, 1970 Edition and the 
Stanford Early Achievement Test . Of these two, the Stanford Early School 
Achievement Test was selected for use in measuring the achievement of first 
grade students in the pretest phase of the experiment, primarily because of 
the large number of readiness items it contains for measuring achievement. 
Another consideration, however, had to do with the design of the normative 
study. The study included more than 8,000 students in kindergarten and more 
than 11,000 students in first grade. A good balance of students from cities 
of over 100,000 population; cities from j. 0, 000-100, 000 population; urban 
areas below 10,000 population; and rural areas was also obtained. Communities 
were selected on the basis of median family income and median years in school 
of persons age 25 and above. In each case the median for the norm group was 
very similar to that of the United States as reported in the I960 census , 

Table 11 at the end of this section presents the reliability es- 
timates, and total number of test items for the Stanford test. The reli- 
abilities reported are acceptable for this grade level in comparison to the 
number of items in each subtest. This test is particularly attractive to 
the beginning first grade student. It is administered in five fairly short 
sittings during which the students answer questions aimed directly at assess- 
ing experience with words, numbers, pictures, and sounds. These provide the 
basis for assessing the attainment of more specific instructional goals at 
a later time. 

The California Achievement Test, Level X, was considered the most 
satisfactory test for measuring end-of-year achievement of first grade students 
The booklet is attractive and the printed pages and other materials are quite 
clear. It has a good balance between readiness and achievement items related 
to vocabulary, comprehension, and number skills, 

6 
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The normative information is sound and is based upon a strati- 
fication of school districts according to a geographic location, 
average enrollment per grade, and community type (determined by size 
and density of the coimnunity) , Data are currently being analyzed, and 
at this time it appears that the groups are properly represented in 
the sample in proportion to their approximate ratio within the total 
population, 

Kuder -Richards on reliability estimates for the California test 
(reported at mid-year) are shown in Table 12 for each subtest. These 
reliabilities reflect a problem with the reading comprehension sub -test 
at this level (r = ,759), However, in the analysis, total reading scores 
are analyzed, consisting of = both. .the„comprehens ion and v ocabulary sub- 
tests. Total reading, then, does have a satisfactory reliability (r *= ,95), 

Grades 2, 3, 7, 8, 9 \ 

The 1970 Edition of the Metropolitan Achievement Test Battery was 
selected as the evaluation instrument for Grades 2, 3, 7, 8, and 9, This 
battery was selected primarily because it was one of the most recently revised 
in terms of content and normative data. 

The format of each test is excellent and the subtests are relatively 
short compared to similar achievement batteries. There is no compromise, 
however, in the coverage of content, and at the higher levels, science and 
social studies are included in addition to the traditional coverage of read- 
ing, spelling, language, and mathematics skills. Subtests included in the 
battery for Grades 2, 3, 7, 8, and 9 are listed below. 

Grade 2 : 

. Word Knowledge 

Word Analysis 
Reading 

Part A: Sentences 

Part B: Stories 

Mathematics 

Part A^ Concepts 
Part B ; Computation 
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, Grade 3 - 

Word Knowledge 
Word Analysis 
Reading 

Part A: Sentences 

Part B* Stories 
Spelling 

Mathematics Computation 
Mathematics Concepts 
Mathematics Problem Solving 

Grades 7.. 8, and 9 

i 

Word Knowledge 
Reading 
Language 
Spelling 

Mathematics Computation 
Mathematics Concepts 
Mathematics Problem Solving 
Science 

Social Studies 

The tests of main interest in the analysis, reading and mathema- 
tics, contain subtests which measure the skills as follows* 

• Word Knowledge measures extent of students 1 reading 
vocabulary 

• Word Analysis measures student's knowledge of sound- 
letter relationships or skill in decoding 

• Reading Measures students ' comprehension of written 
material 

• Mathematics Computation measures students ’ ability 
to compute 

• Mathematics Concepts measures students 1 understanding of 
important mathematical principles and relationships 

• Mathematics Problem Solving measures students ' ability 
to apply knowledge in solving numerical problems. 

The battery offers a good measure of the generalized outcomes of 

education in the United States, The tests included in the battery are of 

high technical quality with the estimates of reliabilities shown below in 

Tables 13 through 16, and the normative study includes many of the largest 

school districts in the country. 
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Certification Tests 



Certification tests were selected using the same criteria as 
for the evaluation tests. These criteria were met for the certification 
tests, with the exception of one of the Grade 2 and Grade 3 reading 

tests (The ETS Survey of Primary Reading Development used as one 
of the certification tests in Grades 2 and 3). In this instance, 
the particular criterion not met was that of having norms based on a 
national standardization sample of students from large metropolitan 
and rural school districts. However, the standardization sample, 
although geographically limited, was heterogeneous in its characteristic 
and thus considered appropriate.* Moreover, the ETS tests did meet all 
other selection criteria, and they were highly rated in a report which 
evaluated existing nationally. standardized achievement tests, prepared 
by UClA's Center for the Study of Evaluation,** 

j 

A listing of all certification tests used is given in 
Appendix J, for each grade and subject area. A description of these 
tests is also given there, covering the same items as previously 
described for the evaluation tests. 



*See Appendix J for a description of the norming sample used. 

**0p. Cit,, page 44. 
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TABLE 11, RELIABILITY COEFFICIENTS AND NUMBER OF ITEMS 
• FOR THE FIRST GRADE PRETEST (STANFORD EARLY 
ACHIEVEMENT) 



Subtests 


Reliabilities 1 


Number of Items 


The Environment 


.82 (Split-half) 


42 


Mathematics 


.82 (Split-half) 


28 


Letters and Sounds 


.89 (Split-half) 


28 


Aural Comprehension 


.77 (Split-half) 


28 



TABLE 12, RELIABILITY COEFFICIENTS AND 
FOR THE FIRST GRADE POSTTEST 
ACHIEVEMENT TEST) 


NUMBER OF ITEMS 
(CALIFORNIA 


Subtests 


Reliability 

KR20 


Number of Items 


Reading Total 


,950 


116 


Vocabulary 


.953 


am B 


Comprehens ion 


.759 


» mm 


Mathematics Total 


.956 


87 


Computation 


,947 


— 


Concepts and Problems 


,904 


— mm 
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TABLE 13 . RELIABILITY COEFFICIENTS AND NtfiffiER OF TEST 
ITEMS FOR THE SECOND GRADE TEST (MET 70, 
PRIMARY I) 


Reliabilities 

Subtests KR20 




Ninnber c 2 Items 


Word Knowledge 


Oh 

CO 

OQ 1 




35 


Word Analysis 


.90 




40 


Reading 


,95 




42 


Total Reading 


,96 




117 


Total Mathematics 


.93 
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TABLE 14 , RELIABILITY COEFFICIENTS 
FOR THE THIRD GRADE TEST 


.AND NUMBER OF ITEMS 
(MET 70, PRIMARY II) 


Subtests 


Reliabilities 

KR20 


Number of Items 


Work Knowledge 


.93 




40 


Word Analysis 


,90 




35 


Reading 


,93 




44 


Total Reading 


.96 




119 


/ 

Mathematics Computation 

/ 


« 

CO 

as 




33 


Mathematics Concepts 


,85 




40 


Mathematics Problem 
Solving 


.88 




• 

35 


Total Mathematics 


.95 




108 
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TABLE 15 , , RELIABILITY' COEFFICIENTS AND NUMBER OF TEST 
ITEMS FOR THE SEVENTH GRADE (MET 70, 
INTERMEDIATE ) 



O 






Reliabilities 




Subtests 


KR20 


Number of Items 


Word Knowledge 


,92 


50 


Reading 


e 93 


45 


Total Reading 


,96 


95 


Mathematics Computation 


.84 


40 


Mathematics Concepts 


.88 


40 


Mathematics Problem 






Solving 


.89 


35 


Total Mathematics 


.95 


115 



TABLE 16 , RELIABILITY COEFFICIENTS AND NUMBER OF TEST 
ITEMS FOR THE EIGHTH AND NINTH GRADE TEST 
(MET 70, ADVANCED) 



Subtests 


Reliabilities 

KR20 


Number of Items 


Word Knowledge 


.93 


50 


Reading 


.92 


45 


Total Reading 


.96 


95 


Mathematics Computation 


,91 


40 


Mathematics Concepts 


,90 


40 

* 


Mathematics Problem 
Solving 


.90 


35 


Total Mathematics 


-.96 

/ 


115 
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TESTING PROCEDURES AND CONDITIONS 



In this section of the report, a summary is given of testing 
procedures and testing conditions for both the pre- and post-testing 
phases. Topics covered include test administration design, selection 
and training of test coordinators, implementation of the testing, and 
test coordinator reports on testing conditions. 

Pretesting 

Administration Design 

The test administration design centered on a program of testing 
which included two minimal conditions as follows, 

t All primary grades were to have the tests administered 
in morning sessions, in class sizes of 35 or less in 
Grades 2 and 3, and a class size of 25 or less in 
Grade i 4 

• All junior high school students were to have the tests 
administered in morning sessions with the exception of 
one separately timed test administered in the afternoon. 

Class sizes were to be 100 students or less, with each 
group having one proctor for every 50 students in addi- 
tion to the test examiner. 

The two administrations, evaluation and payment (or certification) 
testing, were organized consistent with these minimal conditions and a 
testing schedule was devised for each, The evaluation testing called for 
two consecutive morning sessions for the elementary grades and two consecu- 
tive morning sessions plus one test administered on two consecutive 
afternoons for the junior high school grades. Evaluation tests were 
always administered before the certification tests (for both pre- and post- 
testing), The rationale for having the evaluation tests administered first 
was to preclude the possibility of introducing "practice effects" as a 
source of bias in the overall evaluation. The evaluation tests were 
administered to the experimental, control, and comparison groups, and, at 
some sites, to special treatment groups of students. 
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The second test administration, certification or payment testing, 
involved the administration of different batteries of tests to only the 
experimental students in order to obtain test scores which would serve as 
a basis of payment for the technology contractors, Ihe certification test 
batteries were used to measure reading and mathematics skills only. The 
certification pretesting called for tiro consecutive morning sessions for 
the elementary grades and one morning session plus one separately timed 
test in the afternoon for the junior high school grades. One of three 
different test batteries was randomly assigned and administered to one- 
third of the experimental students in each of Grades 2,3,7 ,8 and 9, Only 
one certification test battery was administered to the students in Grade 1. 

Ihe recoranended testing schedules were designed to eliminate the 
effects of fatigue, limited span of attention, and satiation, especially 
in the administration of tests for the elementary grades. The class size 
recommendations previously cited were developed to provide for control of 
the test administration to assure standard conditions, to preclude cheating, 
and to speed the administration of the tests. 

Based on the reconmended class sizes, and the targeted number of 
students to be tested in the various grade /groups at each site, estimates 
were made of the numbers of test examiners arid proctor# required to 
administer the tests at each site, for both evaluation and certification 
testing. For the primary grades, plans were made to recruit these examiners 
and proctors from among certified substitute teachers at the site. For the 
-junior high school level, it was recommended that guidance personnel at the 
site be used to administer the tests, since such personnel are most familiar 
and experienced in test administration. Teachers were to be used in the 
event that guidance counselors could not be recruited. 

In addition, it was planned that the Battelle test coordinators would 
provide a 1-day training session for the test examiners in the procedures 
for administering the tests. This training would be given to assure 
standardization of test administration procedures. Special training materials 
were developed for and piovided to the Battelle test coordinators to assure 
some standardization in the training of test examiners. The training 
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materials. In addition to outlining all aspects of the program and activi- 
ties the coordinators would undertake, included handouts for test examiners 
which outlined their responsibilities and the principles of standardized 
test administration. 

Selection and Training of Test Coordinators 

All of the test coordinators selected by Battelle -Columbus to 
coordinate the test administration at the sites had at least a Master’s 
Degree in psychology or a field of education, and had completed a course 
in tests and measurement. Of the 17 test coordinators who visited 20 
sites, 5 had Doctorates and 12 had Master’s Degrees, Six of the test 
coordinators were full-time Battelle -Columbus staff members, and 11 were 
consultants of Battelle -Columbus , Seven of these consultants are engaged 
full time in testing programs or services involving testing. Thus, all of 
the test coordinators were professionally qualified to coordinate the pre- 
test administration at the project sites. 

The test coordinators for the pretesting phase were briefed 
during a 1-day session at Battelie's Columbus Laboratories, Ihe training 
consisted of an overview of the organization of the experimental program, 
identification of the groups of students to be tested at each site, and 
the specific responsibilities of Battelle -Columbus , Finally, the test 
coordinators were instructed as to the prearrangements made prior to their 
arrival on site, responsibilities they would have on site, and specific 
information needed by them to complete the pretesting arrangements. 

Emphasis at the session was directed toward the establishment of a con- 
sistent pattern of operation among the various testing sites. Such an 
emphasis was necessary to provide as much standardization in the 
administration of the tests as possible. 

w 

Pretest Implementation 

The Battelle test coordinator arrived on site 3 days prior to the 
administration of the evaluation tests to review with the Project Director 
the organizational plans for conducting the test administration, and to 



56 



observe physical facilities. Also, during this time, test materials 
arrived at the site, and the coordinator inventoried materials received 
against materials required. Although in several cases, difficult 
logistical problems were encountered in getting testing materials to the 
sites, the materials arrived at each site in tine and in sufficient 
quantities to test the targeted number of students, with few minor 
exceptions that were rectified limned lately, by sending additional materials. 

Problems concerning the selection and training of test examiners 
at the sites were minimal. Also, test coordinators reported that the 
cooperation of school personnel directly involved in the study was 
excellent, A few instances were recorded in which some of the staff 
members at certain sites presented problems, but none of them seriously 
threatened the pretest administration and most were resolved prior to 
the administration of the tests. 

There were, however, several problems encountered at the sites 
in organizing and implementing the testing program, especially in the 
areas concerning student selection, to a lesser extent arrangement of 
physical facilities, and difficulties in adhering to the testing 

schedule. These problem areas were outlined in a previous report to QEQ*. 
Of concern to this report, however, are assessments of classroom testing 
conditions at the sites, since such assessments bear directly on 
interpretation of the results and conclusions subsequently presented 
. in the analysis section of this report. 



Testing Conditions 



The Battelle test coordinators were asked to report any event that 
occurred during the actual administration of the tests, which, in their 



★•'Pretesting for the Office of Economic Opportunity Performance incentive 
Experiment in Education ,, , January 25, 1971, Battelle^ Columbus 
Laboratories, Colimbus, Ohio, 43201. 
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opinion, might endanger the validity of test results. In this reporting, 
they were asked to be specific about grade level and testing- group 
whenever necessary, A summary of these reports is given below, by site, 
for the evaluation pre-testing. 

Grand Rapids and Bronx, At two of the 18 sites (Grand Rapids and 
Bronx) reported events and testing conditions during the pretesting were of 
such a nature that a decision was made by Battelle and 0E0 to retest certain 
grade /groups, in order to obtain more valid test data. At Grand Rapids, 
severe reservations were expressed by the site Project Director and the test 
coordinator concerning the validity of Grades 7, 8, and 9 evaluation 
testing due to discipline problems, testing facilities, and difficulty 
of students in hearing test administration directions in the facilities 
used. Consequently, it was decided to retest these students, and the 
second testing went smoothly. 

At Bronx, the second testing effort was much more extensive, 
since a greater number of grades and testing groups experienced unusual 
circumstances during the original testing. Retesting occurred for all 
grade/groups except Grades 1, 2, and 3 control groups. Retesting was 
judged essential due to a combination of several factors, including 
inability to get students identified for testing, inadequate and crowded 
physical facilities, humid weather, and student fatigue and disruptive 
behavior. 

Conditions at Bronx were greatly improved on the second testing. 
There was a problem, however, in that the number of students tested in 
the Grades 7, 8, and 9 control groups was less than the planned number, 
on the retesting, 

i 

Reports of evaluation pretesting conditions for the other 16 
technology company sites are sunmarlzed in Table 17, As can be seen from 
the table, for Anchorage, McComb, Rockland, Selmer, and Taft, no incidents 
were reported, for any grade, group, or subject matter area. 
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TABLE 17, TESTING INCIDENTS REPORTED FOR PRE- EVALUATION TESTING 



Site 




Incidents or Conditions 


Anchorage 


• 


None reported 


Athens 


• 


Grades 7, 8, 9, most experimental 
groups, reading: discipline problems 


Bronx 


0 


None reported on retesting 


Dallas 


0 


Junior High: Organizational confusion 
in getting testing underway, and in 
selecting students, that might have 
affected students test taking behavior 


Fresno 


m 


Grades 7, 8, 9, experimental and 
control, reading and mathematics : 
discipline problems, students not 
paying attention, some testing groups 
larger than planned 


Grand Rapids 


m 


Grades 2 and 3, control, reading and 
mathematics, for approximately 20% of 
students: noise, cheating 




• 


No incidents reported on Grades 7, 8, 9 
retesting 


Hatranond 


0 


Junior High: Low motivation and difficulty 
in getting students interested 


Hartford 


0 


Elementary and Junior High: Discipline 
problems; not following directions well; 
tendency of students to put little 
effort into the test taking 


Jacksonville 


0 


Grade 7, experimental, reading and 
mathematics: poor handling of group by 
examiner; noisy outbursts, test marking 
without question reading 





5? 

TABLE 17. (Con tinned) 


Site 


Incidents or Conditions 


Las Vegas 


9 Grades 1, 2, 3, experimental and control , 
reading and mathematics (more than 50% 
of students) : crowded conditions and 
testing of two or three grade levels 
in single large room; discipline 
problems and controlling students; 
noise and talking from students 

• Grades 8 and 9, experimental and control, 
reading and mathematics* continued 
talking; discipline problems 


Me Comb 


• None reported 


Philadelphia 


« Extreme heat (95 F) during testing; rooms 
not air conditioned 


Portland 


• Grades 7, 8 , 9* experimental and control, 
reading (all students) * difficulty in 
controlling students; over crowding in 
testing stations 


Rock! and 


• None reported 


Seattle 


• Grades 7, 8 , 9, reading and mathematics; 
Discipline problems; disruption of one 
testing period by a false fire alarm; 
confusion during testing 


Selmer 


• None reported 


Taft 


m None reported 


Wichita 


• Elementary students; restless, un- 
motivated students; extensive testing 
previous to project testing; both days 
of evaluation testing compressed into 
one day; unskilled test administrator 
for two third -grade control groups 
(36 students .total) 

0 Secondary students; unruly, disinterested, 
unmotivated ; some students marking answers 
at random 
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For five other sites (Dallas, Hammond, Hartford, Philadelphia, 
Wichita), incidents and/or test conditions were not reported as specific 
to a grade, group, or subject matter level. Rather, certain general 
conditions were cited for either the elementary level, junior high level, 
or both levels. Thus, at Philadelphia, extreme heat (95 degrees) was 
reported during all testing sessions, and none of the testing rooms was 
air conditioned. At Dallas, certain organizational problems were cited 
at the junior high level, particularly difficulties in identifying the 
target population to be tested (several srndents were tested who were 
later identified as not being in the program, plus a high absence rate 
at the time of testing for those identified as being in the program) , 

In turn, it was judged that these problems might have affected students' 
behavior during actual tasting. How many students might have been 
affected is unknown. 

In Hammond, Hartford, and Wichita, test coordinators reported 
problems concerning the maintenance of student discipline, and the 
test-taking motivation of students. At Hammond, this was reported for 
the junior high students. At Hartford, the coordinator reported that 
"testing sessions appeared to be conducted as efficiently as possible 
considering the type of youngster being evaluated; that is to say that 
the students in this project in this system were rather undisciplined, 
did not follow directions well, and had a tendency . tQ .put little effort 
into this total evaluation process". At Wichita, both elementary and 
secondary students were reported as being "turned-off" to the whole 
situation, as manifested in restless and sometimes unruly behavior, and 
manifested by some students at the secondary level sleeping through the 
test or marking answers on the tests at random. At the elementary level, 
contributing factors at Wichita were the reported extensive testing of 
students prior to the project testing, and the compressing of the 
testing into one* day, rather than spreading the testing over two days 

y 

as planned. 

For the remaining six sites (Athens, Fresno, Jacksonville, 

Las Vegas, Portland, Seattle) incidents were reported specific to a 
given grade, group (E or C) , and subject matter combination. In Table 17, 
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for these six sites, the grade (s), group (s) , and subject area(s) for 
which the incidents are reported are indicated first, followed by a 
brief description of the incident (s) . Also given, when available, is 
an estimate of the percentage of students within a given grade (s), 
group(s), and subject matter area(s) for which the incident pertains. 

As can be seen in the table, for these six sites, again 
problems centering on discipline and lack of test-taking motivation 
were the type of incidents reported, with these reports occurring at the 
junior high level. In addition, however, for certain of these sites, 
as shown by the table, conditions of overcrowding were reported, and 
also the testing of groups at a testing session in larger numbers than 
planned. In turn, these conditions probably accentuated to some extent 
the discipline problems and lack of student interest in taking the test. 

Also, at one site, "poor handling" of one of the grade 7 testing groups 
was a factor in the reported discipline problems. At another site, the 
testing of two or three grade levels (at the elementary) in a single 
room no doubt contributed to the problems and, at a third site, student 
behavior would probably have been improved had the testing effort been 
better planned at the site. 

In summary, in those 13 sites where general or specific 
incidents or conditions were reported, the type of problem reported most 
often centered around discipline problems in the classroom and lack of 
test- taking motivation*. These problems were limited generally to the junior 
high level, and in several cases also limited to only some of the grades, 
schools, and testing groups within this level. In certain cases, these 
problems were accentuated by potentially "controllable" factors, such as over 
crowding in the testing rooms, number of students tested at a sitting, 
amount of testing done in one day, etc. The latter implies that a re- 
testing of certain sites might have yielded better testing conditions 
(as with Grand Rapids and Bronx), However, decisions to retest had to 



* To a lesser degree, problems were created by administrative 

confusion resulting from the opening of school, late selection of 
j students to be tested, and sometimes lack of communication to building 

; personnel, 

i 
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be based on several factors and considerations, not the least of which 
were budgetary considerations, judgments as to numbers of students possibly 
affected, and, quite importantly, for a given site the feasibility and 
acceptance by the site of further disrupting students' learning schedules 
by yet more testing. Moreover, it seems clear that the general problem 
centers more on the students who were tested, rather than on "test 
conditions", especially at the junior high level. The performance 
incentive experiment is concerned with students many of whom have lost 
faith in the school program and are simply not motivated to do well in most 
school functions, including their performance on standardized achieve- 
ment tests. In turn, if youngsters were "turned— off" on a first testing, 
a second testing shortly following might' have yielded even more student 
disinterest and behavioral problems. 

Moreover, several of the types of incidents and conditions 
reported for certain sites are not uncommon to any large scale stand- 
ardized testing programs. The effect that such incidents have on test 
results, combined with the more unique problems in this study of student 
motivation and discipline for this target population, is not known. 

However, it is important that these circumstances be as explicitly noted 
as is possible, for they do serve to qualify the results and conclusions 
subsequently made in the last section of this report, at those sites, 
grades, and subject areas where incidents were reported. 

Posttesting 



Planning 



Experience from the pretesting effort provided valuable back- 
ground for facilitating an organized and smoothly running posttesting 
program. Advance planning by personnel at Battelle resulted in several 
procedure and scheduling changes that virtually eliminated the administra- 
ttve, student identification, and fatigue problems. Changes which proved 
instrumental in overcoming some of the pretesting problems were 

(1) A Battelle representative visited each site for a 

1-4 day period approximately 1 month before testing 
was to begin. The purpose of this planning visit was 
to meet with the Project Director and his starf. 
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building principals, technology company representa- 
tives, teachers, and any other necessary parties to 
specify and agree upon schedules and responsibilities 
for the entire posttesting effort. 

(2) Suggested testing schedules were altered to avoid testing 
on Fridays and Mondays. Ihis gave experimental students 
a 4 to 5-day break between evaluation and certification 
testing and enabled larger numbers of students to be 
tested because Friday and Monday have proved to be high 
absentee days in most schools. 

(3) Tests arrived on site with all student ID information 
already on the booklet. As a result, no confusion could 
arise as to which students were to be tested and which 
tests each student should take. 



Implementation 

In general, posttesting went extremely well in all the sites. 

The number of students tested was very high, and reports on testing condi- 
tions from the Battelle coordinators were favorable. Logistics posed few 
problems, the same high standards were met for recruiting qualified 
coordinators, materials were well prepared, and detailed plans for each 
site were available for the coordinator to study before he arrived on site. 

Testing Conditions 

As with the pretesting, Battelle coordinators were asked to 
report events which, in their judgment, might have endangered the validity 
of test results. These incidents are summarized in Table 18, which 
present incidents In the same format as given previously for the pre- 
testing. 

Although posttesting incidents centered around problems of 
discipline and test -taking motivation, there were somewhat fewer of 
these kinds of incidents, and they were generally less severe. In 
the three situations where problems were judged unusual, the testing 
was stopped and resumed only when conditions had improved. 



ERIC 



! 




64 



In general, in context of the entire posttesting effort, 
those incidents that did occur were relatively small in number and 
often minor in nature, and tended to be localized to few testing groups 
However, as with the pretest, circumstances and conditions noted in 
Table 18 serve to qualify conclusions and results in the analysis 
section of this report, at those grades, sites, and subject areas 
where incidents were reported. 
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TABLE 18. TESTING INCIDENTS REPORTED FOR POST EVALUATION TESTING 





Site 




Incidents- or Conditions 


Anchorage 


m 


Grades 1 and 3, some experimental groups, 
reading and math: inadequate time, 

hurried directions, no breaks between 
tests . 


Athens 


« 


Grades 7, 8, 9, control: minor 

discipline problems, overcrowded 
cafeteria. 


Bronx 




Grade 9 , control, not tested 




m 


Grades 7, 8, 9, experimental and contii Is 
poor motivation, unruly students, par- 
tioned ballroom used for testing. 


Dallas 


m 


Grade 1, experimental, reading, one 
class (approx, 25% of students) : 
hyperactive students during evaluation 
reading tests; regular teacher quieted 
them but minimal attention during rest 
of evaluation test. 


Fresno 


+ # 


Grades 7, 8, 9, experimental group, 
reading: students refused to respond 

to substitute teachers as test 
administrators; and severe discipline 
problems were encountered; However, 
retesting was accomplished. 


Grand Rapids 


• 


None reported. 


Hammond 

ft 


• 


Grade 2, control, math: time limits 
for math subtests severely violated, 
for approximately 20% of students. 

* 
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TABLE 18. (Continued) 



Site 




Incidents or Conditions 


Hartford 


• 


7 th grade j control, reading : schedule 
and time limits violated for one class 
(approx, 50% of students) . 


Jacksonville 


• 


7th grade, experimental, reading: 
crowded room, uncomfortable folding 
chairs, subpar handling by test 
administrators 


Las Vegas 


• 


None reported. 


McComb 


• 


None reported. 


Philadelphia 


• 


Grade 3, experimental, reading and math: 
outside noise, poor moral due to 
cancellation of field trip because of 
tests 


Portland 


• 


None reported. 


Rockland 


• 


None reported. 


Seattle 


• 


Grades 7 , 8 , 9 , experimental : crowded 
rooms, disruptions during testing, poor 
discipline duo to conflicts between 
company staff and regular school staff. 


Selmer 


• 


None reported. 


Taft , 


• 


None reported. 


Wichita 


• 


Grade 2, experimental: one class 

extremely unruly due to lack of control 
by regular teacher. 




• 


Grades 7, 8, 9, experimental: several 

small groups, experienced discipline 
problems due to inability of proctors/ 
examiners to maintain discipline. 
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TIE DATA -ANALYSIS METHOD 



The purposes of this section are to sunmarize the methodo- 
logical issues studied during the selection of the method of analysis, 
to present alternative analysis methods examined, and to present a 
description of the method selected, A brief statement is given of 

/ i = 

each methodological issue studied, and its effect on the selection of 
the method of analysis is delineated. The description of the selected 
method includes a specification and discussion of the analysis models, 
and associated statistical tests employed to obtain the principal 
results of the analysis. 

Issues in the Selection of an Analysis Method 

A number of recognized methodological issues relevant to the 
analysis of the outcomes of this program were studied. Many of these 
issues arose from typical, unavoidable difficulties associated with 
"real world" implementations of experimental designs. Other issues were 
concerned with the identification of a suitable methodology for data 
analysis. There were three subject areas which were judged to include 
those issues relevant to the selection of an appropriate analysis 
method ; 

• Measurement of change 

• Nonrandom sampling 

• Measurement error, 

A breif description and discussion of these areas follows. 

Measurement of Change 

Despite the fact that much controversial literature has been 
generated within the social sciences on the proper methods for measuring 
change (e,g,, change from pretest to posttest), very little is of actual 
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relevance to the problem of this experiment. For this experiment, the 
important, underlying question is not whether to measure “change" from 
pretest to posttest, per se, nor is it whether to use a "gain" or 
"change" score as the dependent variable. Rather, the basic question 
is how to take into account appropriately differences between the E 
and C groups on the pretest and other variables potentially related to 
posttest scores, when comparing the two groups on posttest performance. 

In the analyses described in this report, the pretest and other selected 
variables are taken into account by incorporating them into the analysis 
models as independent variables. 

Nonrandom Sampling 

Probably the greatest number of methodological issues resulted 
from the inability to obtain a random sample of students, schools, or 
groups from some well-defined population. Because of nonrandom sampling, 
an attempt was made to identify and measure several important variables 
upon which the experimental and control groups could be different. For 
example, several important student variables were measured, such as pre- 
test score, race, and several student family background variables (see 
the section entitled "Description of Target Population"), However, 
other important student variables were not measured, such as student 
motivation. Also, the measurement and analysis of E and C school 
differences on such variables as teachers' attitudes toward the experi- 
mental program and the schools' curriculum outside of the experimental 
program were not within the scope of the study. Such differences in 
student and school variables offer competing explanations for the 
observed effects of the remedial program, 

A partial solution to this problem was accomplished by means 
of quantitatively adjusting the observed effects of the remedial program 
on the basis of differences between the groups for some of the variables 
that were measured. The extent to which this was carried out appropriately 
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is partly a function of the limited sample sires at each grade/site/ 
subject combination. Also, the choice of an appropriate model that 
includes several of these variables on which the groups might differ 
and that appropriately represents the functional relations between 
these variables is difficult — and again, even more difficult due to 
limited sample sizes. 

However, group differences on some of the more important 
variables were taken into account quantitatively in the analyses. 
Variables that were taken into account were pretest, student race, 
father's education, total family income, and an attitudinal item from 
the Parent Questionnaire concerning parents' approval of new teaching 
methods for their children. These particular variables were chosen 
because they were judged to have a potentially high re3,ationship with 
posttest performance, because there were observed differences between 
the E and G groups on these variables at many grade/site combinations 
(see the section on "Description of Target Population") , and because 
these variables were judged to provide sufficient data within most 
grade/sites to warrant inclusion in any analysis. Father's occupation 
was considered also but was excluded due to the relatively low response 
rate to that item. 

Differences on other variables measured in the study were 
taken into account only qualitatively. Because group differences were 
judged to exist on these other measured variables, and because group 
differences may exist on variables not measured, the conclusions 
presented are appropriately qualified. 

Also, because nonrandom sampling creates uncertainty about the 
actual sampling distributions of commonly calculated statistics, such as 
"t", "F", and chi-square, the validity of statistical tests of signifi- 
cance is threatened. That Is, the probability formally associated with 
any one of these statistics taking on an extreme value is actually 
unknown and can only be taken as an approximation. In the following 
analyses, the t-stacistics calculated are only "formal" calculations. 

The probabilities associated with these calculations are assumed to be 
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approximately correct. Usually , a t -value greater in absolute value 
than 2,0 is taken as evidence for a nonrandom result in all tests for 
significance. In some analyses, the limited sample sizes require a 
higher t -value in order to maintain the significance level at approxi- 
mately 0,05, 

Measurement Error 



The assumption that independent variable values do not repre- 
sent hypothesized "true” values raises the issue of '‘measurement error" 
in the analysis model. The presence of measurement error in such 
variables as pretest and posttest scores is usually indicated by 
estimating the reliability of such tests. The pub Ushers' estimates of 
the reliability of the achievement tests used in this experiment are 
discussed in an earlier section. These estimates are consistently high. 
However, in a study dealing with a target population of low-achieving 
students from low-income families, reliability estimates based on samples 
from such a population are also important. Such estimates were obtained. 
Table 19 presents reliability estimates using an internal consistency 
measure (Kuder -Richardson 21), As can be seen from the table, all of 
these estimates are quite high, the lowest be 5 * tg 0.90. 

These reliability estimates were obtained using the Kuder- 
Riehardson 21 formula for the following reasons. An internal con- 
sistency estimate was judged most appropriate because the design of the 
study did not provide for obtaining additional test data at any other 
times than the beginning and end of the 1970-71 school year and the 
beginning of the 1971-72 school year. Thus, reliability estimates 
involving fluctuations in a student's score oyer a short time interval 
as a source of error, such as the correlation of alternate forms 
administered 2 weeks apart, were ruled out. The best estimate of 
reliability based upon an internal consistency measure is that most 
commonly referred to as the Kuder -Richards on 20 formula. All reliability 
estimates involving the correlation of variously defined halves of a 
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TABLE 19, ESTIMATED RELIABILITY COEFFICIENTS (KUDER -RICHARDS ON 21) 
OF THE ERE- AND POSTTEST FORMS OF THE EVALUATION READING 
AND MATHEMATICS TEST FOR EACH GRADE (a) 



Reading 
' KR-21 



Mathematics 

KR-21 



Grade 1 

Pretest (SEAT, Level I)^ C ^ 
Posttest (CAT, Level I, Form B) 



0,94 2139 

0.90 2139 



0.94 2124 

0.92 2124 



Grade 2 



Pretest (MAT, Primary I, Form F) 
Post test (MAT, Primary I, Form G) 



0,92 2702 

0.98 2702 



0,88 2531 

0,97 2531 



Grade 3 



Pretest (MAT, Primary II, Form F) 
Posttest (MAT, Primary II, Form G) 



0,94 2482 

0.96 2482 



0,98 2357 

0.94 2357 



Pretest (MAT, Intermediate, Form F) 
Post test (MAT, Intermediate, Form G) 



0,93 2319 

0.94 2319 



0.93 2286 

0.95 2286 



Grade 8 

Pretest (MAT, Advanced, Form F) 
Posttest (MAT, Advanced, Form G) 



0,91 2256 

0.92 2256 



0.90 2153 

0.93 2153 



Grade 9 

Pretest (MAT, Advanced, Form F) 
Posttest (MAT, Advanced, Form G) 



C .93 2089 

0.93 2089 



0,94 2077 

0.94 2077 




(a) The sample used to estimate KR-21 were full-year students with both a 

pre- and posttest score in the appropriate subject, 

* * 

(b) N = the number of students in each sample, 

(c) See an earlier section of this report for more complete identification 
and discussion of each test. 
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test, such as "split -half” and "odd -even", provide estimates of the 
value calculated using the Kuder -Richards on 20 formula. In fact, the 
average of all correlations between all possible halves of a test is 
equal to the Kuder -Richards on 20 value. However, the Kuder -Richards on 20 
formula was not used to estimate reliabilities because it requires that a 
pass -fail response to each item for each student be stored in the data 
file. Such storage was not within the scope of this study. However, if 
one assumes that the item difficulty (percent who passed an item) of 
each item is equal to the average item difficulty, the Kuder -Richard son 20 
formula reduces to a formula involving only the number of test items, and 
the mean and variance of the raw scores for the students in the sample. 
This formula is most concnonly referred to as the Kuder -Richardson 21 
formula. Finally, it is always true that this formula will provide a 
lower estimate of a test's reliability than the Kuder -Richardson 20 
formula. This fact makes the high test -re liability estimates obtained in 
this study even more impressive. 

In Batte lie -Columbus ' Interim Report on this experiment, quali- 
tative judgments were made about the pretests' reliabilities in order to 
qualify any analysis results which might be suspect due to potentially 
unreliable pretest scores. These judgments were made on the basis of 
the correlations between pretest and posttest scores within the 1 and C 
groups at each site. At this time, however, the high estimated reliabili- 
ties in Table 19 are most encouraging and obviate the necessity for such 
judgments. It is still encouraging, however, to note that the large 
majority of these pre-post correlations are relatively high, as shown in 
Table 20. Table 20 provides a frequency distribution of the pre-post 
correlations*, by grade and subject. This table shows that the values 
of the pre-post correlations increase by grade level in the elementary 
grades and are generally high (above 0,70) in the’ junior high grades. 



*See Appendix K for the values of the pre-post correlations within E and 
C groups, for each grade, site, and subject area. 
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TABLE 20. DISTRIBUTION OF PRE-POST CORRELATIONS FOR EXPERT - 

MENTAL AND CONTROL GROUPS BY GRADE AND SUBJECT AREA 



Value 


Grade 1 


Grade 2 


Grade 3 


Grade 7 


Grade 8 


Grade 9 


of r 


R 


M 


R 


M 


R 


M 


R 


M 


R 1 


M 


R 


M 


.00- .10 










. 


. 




. 








. 


.11- .20 


* 


- 




- 


1 


- 


- 


- 


- 


- 


- 


- 


.21- .30 


3 


1 


- 


- 


- 


- 


- 


- 


1 


1 


- 


- 


,31- ,40 


1 


3 


3 


- 


1 


1 


1 


2 


1 


1 


- 


- 


.41- .50 


6 


8 


5 


2 


4 


2 


- 


2 


1 


3 


1 


1 


.51- .60 


10 


5 


16 


8 


4 


2 


2 


3 


5 


3 


1 


2 


,61- .70 


6 


11 


9 


12 


9 


11 


8 


4 


6 


7 


1 


5 


,71- ,80 


7 


6 


3 


14 


16 


14 


11 


11 


14 


10 


12 


11 


,81- .90 


1 


- 


« 


— 


1 


6 


12 


14 


5 


10 


17 


11 


,91-1.00 


** 


- 


— 


■* 


— 


- 


2 


« 


3 


1 


2 


4 


Totals : 


34 


34 


36 


36 


36 


36 


36 


36' 


36 


36 


34 


34 



R =s Reading 
M * Mathematics 

The correlations at Grade 1 are understandably lower because the pre- and 
posttests were not alternate forms of the same test. The pretest was a 
readiness test and the posttest was a standardized achievement test. In 
reading at Grades 2 and 3, most correlations are above 0.50 and 0.60, 
respectively. In mathematics at Grades 2 and 3, most are above 0.60 and 
0,70, respectively. Overall, very few correlations are less than 0,40, 

Selection of an Analysis Method 

Five methods were considered as candidates for the analysis of 
the data in this study: 

• compare the difference between the E and C post- 
test means with the difference between the E and 
C pretest means. 
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• assume a randomized blocks experimental design with 
pretest score as the blocking variable and analyze the 
results using the analysis of variance, 

• assume a completely randomized experimental design with 
pretest score as a covariate and analyze the results 
using the analysis of covariance, 

9 compare the post test scores for the E group with pre- 
dicted scores for the E group obtained from a 
regression model fitted to the data for the C group, 

• treat posttest score as a dependent variable; treat 
group membership, pretest scores, and selected other 
variables as independent variables, and analyze the 
results using regression analysis. 

The first method was judged unacceptable because it does not pro- 
vide a quantitative adjustment in mean posttest differences due to mean 
pretest differences and does not consider the selected other variables at 
all. The second method was not considered appropriate because the blocking 
variable is the pretest score and it may interact with the group variable. 
These conditions make ,, blocking ,, undesirable. The third method can account 
for a linear relationship between pretest and posttest, but it requires 
assumptions analogous to the block design that do not permit the study of 
interactions. The fourth method was judged unacceptable because it does 
not use the data from the E group to fit the regression model. 

The fifth method can account for linear and nonlinear relations, 
and it allows the inclusions of interactions of pretest and other variables 
with the group membership variable. Group differences on the posttest may 
then be examined with the understanding that they have been adjusted for 
differences on the pretest and other variables and that any necessary inter- 
actions have been included in the analysis model. For these reasons, this 
method was selected for the analysis of the data. 
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Regression Analysis Models Employed 



Two types of regression models were employed to provide the 
principal results. One involved group membership, pretest score, and 
their interaction as independent variables (called the Pre-Post Model) , 

The second included pretest score and the family, background ar.d at- 
titudinal variables discussed earlier without any interaction terms 
(called the Extended Variables Model}. . . 

Both of these models take into acr mt only linear relations 
between pretest and posttest. Examination was made of the appropriateness 
of using only linear functions of pretest scores. This was accomplished by 
examining 236 computer -genera ted scatter plots. Each scatter plot repre- 
sented each grade/site/subject combination, illustrating the actual rela- 
tionship between pretest and posttest. Study of these plots indicated that 
nonlinear terms involving pretest were unnecessary. However, a few of the 
scatter plots at the elementary level did indicate some curva linearity, in 
that there appeared to be, in these cases, a smaller rate of change (or slope) 
of posttest scores on pretest scores for the higher values of pretest. In 
most of these cases, the grade/sites involved were those where the pretest 
scores were overall relatively high and the higher scoring students on the 
pretest did not have as much room to improve (a ceiling effect) as the lower 
scoring students. 

An example of a typical scatter plot for each grade is presented in 
Appendix L, 



The Pre-Post Model 



The data for each grade/site combination were fitted for 
the subjects Reading and Mathematics scores using a regression model of the 
following form: 



Yf - A + B 1 X li + B 2 X 2i + B^.X,. + e t 
where A, B^, Bg, B^ denote regression coefficients and 



Y. « posttest score for student i, i = 1, ,,,, n; 

X- ^ « 1, if student i belonged to the experimental group, or 

0, if student i belonged to the control group; 
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. = pretest score for student i ; and 

e, a* deviation between the actual posttest score and the 

estimated posttest score using the fitted regression model. 
Substitution of X u * 1, 0 into the model yields 



and 




(A + + (B 2 + B 3 ) X 2 . + e u 




A + 



> X 2i + 



Oi 



as the individual regression lines fitted to the experimental and 
control data, respectively* The Pre-Post Model was fitted to the data 
using the SPSS* regression program as implemented on Bat telle -Columbus 
Laboratories' CDC 6400 computer* The difference between these 
regression lines after they have been estimated from the sample data** 
is then given by 



Y 0 - B 1 + B 3 X 2' 



This expression shows that the difference between the regression estimate, 
Y , of the posttest score for experimental students and the regression 

mm ^ 

estimate, Y n , of the posttest score for control students is given by 
a linear function, + B^X^, where X^ denotes a given specified 
pretest score. This is to say that the estimated difference between 



the two groups on the posttest varies for different values of pretest. 
Such variation is a linear function of the pretest score in this model. 

Figure 2 provides a graphic illustration of this regression 



model. The difference between estimated posttest scores for the two 
groups is the difference between the two regression lines evaluated at 



any pretest level* Two such differences are described. One is the 
difference at the point where the pretest score is Eero, This difference 
is algebraically equal to B-, the estimated coefficient of the group 
membership variable. Its magnitude and sign answer the question "What 
is the estimated difference between the groups when the pretest score is 
zero?" Such a question warries little substantive meaning, A more meaning 



* Statistical Package for the Social Sciences, Nie, N, , Bent, D.H, , Hull, 
C, H, , McGraw-Hill Book Company, New York, New York, 1970, 

**The carat above each letter indicates that it has been estimated from 
sample data. After estimation, the subscript, i, is dropped. 
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ful question is to as|c what the difference between the groups is estimated 
to be for various values within the pretest score range for which data exist 
in both the E and C groups. An example of such a score range is illustrated 
in Figure 2 by the two vertical broken lines. Note that, because of the 

/k 

difference in slopes of the two lines (measured by B^) s the difference between 
groups can change in magnitude and direction as .the pretest level changes. 

For example, the group difference favors the C group when - 20, but 
favors the E group when - 60, 




0 (0 20 30 40 50 60 TO 60 90 

Pretest Score (Xg) 



FIGURE 2, ILLUSTRATION OF THE "PRE -POST MODEL" 
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e Method Employed for Obtaining Principal Results With the Pre - 
Post M o del . In order to assess the impact of the experimental program for 

, , A A 

any g 



site/sub jec. t , the difference, Y^ 



was evaluated at the 



mean pretest, X^, taken over both the experimental and control students 
within each grade /site /subject combination. Each resulting difference 
was then divided by its estimated standard error, 8 q, to obtain a ratio 
having the form of a t-statistic* Values of t greater than 2,0 were 
then taken as evidence for a significant group difference. 

The standard error of the difference, s^, was obtained as 

A A A A 

follows. Because Y^ - Yq is a linear combination of and it 
may be shown that 



2 -2 2 

sf + X«s„ + 2X«s 



2 3 



2 * 13 ' 



where 



A denotes the estimated variance of Y, 
0 I 



Yq evaluated at 



2 2 a a 

s„ and s„ denote the estimated variances of B_ and B, 
13 1 ; 



respective! 



and 



denotes the covariance between B, 



A 

B„ » 



AO 

Values for s^ and were obtained from the computer printout of the regression 
Values for the covariance, s^ 3 , were obtained using the formula* 



where 



13 



< nr 13 s Xl s x 3 > 



r 12 r 23 r 13 



13 



1 + 2r 12 r 23 r 13 - r 12 



13 



s denotes the residual mean square; 

r i 2 , r 13 , r 23 denote the estimated correlation coefficients 
between group (X^) and pretest (X 2 ) , between and the group- 
by-pretest interaction (X^X 2 ) , and between X 2 and XjX 2 , 
respectively; 

s , s denote the estimated standard deviations for X^ and X^X 2 ; 
X 1 x 3 

and n denotes the number of students. 
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This particular formula for calculating was used because the values 

on the right-hand side were available from the computer printout. It 

2 • 

was derived from the relation = c ^ 3 ® where denotes the (1*3) 

element of the inverse of the matrix of sums of squares and cross- 
products of deviations for X^ , X^, and X^, 

To illustrate this t-test, one can draw the fitted regression 
lines for the E and C groups and evaluate them at the combined E and G 

A A 

mean pretest score. The difference between the lines, Y^ - Yq, at this 
point is the difference tested for significance. In the following 
sketch: 




X^ represents the combined E and C mean pretest score, Y. , the estimated 
mean posttest score for the experimental program, is shown by the mark 

A 

on the E line; Y^, the estimated mean posttest score for the traditional 

program, is shown by the mark on the C line; and the difference between 

A A 

the two estimated means (Y^ - Y^) is shown by a vertical line between 
the two marks. This estimated posttest difference (at the combined 
E and G mean pretest score) forms the numerator of the t-statistic 
employed to test for significant group differences. This test forms 
the basis for the findings concerning experimental program effects in 
this study. If this t-test yields a value greater than +2,0, this can 
be taken as evidence that, in a larger collection of students (at the 
particular grade/site) with a pretest score distribution similar to the 
combined E and C pretest score distribution observed in the sample, 
the mean posttest score for those students if given the experimental 
program would be higher than the mean posttest score, of those students 
if given the traditional program. Of course, if this t-test yields a 
value less than -20, one would predict a lower mean posttest score for 
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students if given the experimental program than if given the traditional 
program. 

In the above sense , the t-cest employed provides an assessment of 
the ’’overall 1 * effect of the E and C treatment conditions qualified, as 
mentioned earlier, on the basis of group differences on variables other 
than pretest. The outcome of this t-test, however, is not qualified if 
group differences exist on pretest . This is because the estimated means 
for the experimental and traditional programs are generated at a given 
pretest value, or for a given pretest distribution, so that any differences 
between the two groups in estimated posttest means cannot be attributed 
to group differences on pretest. In contrast, any difference between the 
1 and C groups on their actual posttest means can be attributed partially 
to differences between the groups in their initial entry level. Thus, 
the regression technique, in the above outlined way, "takes into account" 
or "adjusts for" initial differences between the groups on the pretest. 

This 1 concept is extended to adjust for group differences on variables in 
addition to pretest, in the Extended Variables Model, discussed subsequently 
in this section. 



Study of the Significant Group-by-Pretest Interactions . The 
"Pre-Post Model" fitted to the data at each grade/site/subject combination 
includes a group-by-pretest interaction Jn the slopes of the E and C 
posttest regression lines as functions of pretest scores. If such a 
difference in slopes exists, then the inclusion of this term will result 
in an improved estimate of the residual error. This improved estimate, 
in turn, will yield a better statistical assessment of the difference 
between the E and C regression lines evaluated at the mean of the pre- 
test scores for the combined E and C groups. This latter assessment is 
taken to be the primary measure of the impact of the' experiment at each 

grade/site/subject combination, as mentioned above, 

* 

The inclusion of the interaction terms in the model also yields 
secondary benefits. Whenever the interaction is significant, the 
regression lines will have markedly different slopes, and may even 
intersect in the range of the pretest scores. This has the effect of 
showing reversed differences between the fitted E and C posttest means 
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corresponding to low and high pretest scores. Because of the importance 
of this kind of information, a suiwnary of the results obtained at all 
those grade/site/subject combinations which yielded a group-by-pretest 
interaction coefficient having a t -value greater than 2,0 is presented in 
Appendix M e 

The Extended Variables Model 



The data for each grade/site combination were also fitted, 
wherever possible, using a regression model with linear terms of the form: 



Y, 

x 



A + B-X- , + B 0 X« , + B„X_ , 
1 li 2 2i 3 3a. 



+ B X , + e, 
p pi x 



where A , 



Bi ,* * *, B denote the regression coefficients and 

i * * p 

Y. * posttest score for student i, i = 1, • • •, n; 

X,,« 1, if student i belonged to the experimental 
lx 

group, or 0, if student i belonged to the 




control group; 
pretest score for student i; 




e. 

x 



**,Xp^ = total family income, father's education,* 
race, and parent's response to the 

< 

"Approve New Method" attitude item, for 
student i; and 

deviation between the actual posttest score and the esti- 
mated posttest score using the fitted regression model. 



As in the Pre-Post Model, evaluating X.. = 1 yields the model for 
the experimental group and X.. = 0 yields the model for the control group. The 
difference between these two regression models after they have been esti- 
mated from the sample data is then simply 



/V 

Y, 



/s 

Y, 



O’ 



* Father's education is divided into three levels (as defined in the 

section on "Description of Target Population") and, thus, requires two 
zero-one "dummy" variables. Race requires either none, one, or two 
zero-one "dummy" variables, depending upon whether a grade/site has 
one, two, or three predominant races. 
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That is, B^ is the estimated E/C difference in posttest means for any 
given value of pretest, or for that matter, for any given combination 
of values of the independent variables including the family background 

A 

variables and the attitudinal items. Thus is an estimate of experimental 
program impact holding constant the other variables included in the model, 
so that the value of (the estimated difference in posttest means between 
experimental and traditional instruction) cannot be attributed to group 
differences in these other variables. 

The group-by-pretest interaction term was not included in the 
Extended Variables Model because E/C group differences estimated with the 
Pre-Post Model were found to be quite similar (both in magnitude and 
statistical significance) to such differences in supplementary regression 
analysis performed at all grade/sites using a model with only group and 
pretest as independent variables . Overall, more than 90 percent of the 
statistical outcomes of the Pre-Post Model, comparing the E and C groups, 
remained unchanged when the interaction term was dropped in the supple- 
mentary analyses. This is not surprising when one considers that despite 
the presence of an interaction in the Pre-Post Model, the group comi arisen 
was made at the combined E and C pretest mean. Thus, for example, using 
the Pre-Post Model with data that demonstrated a strong cross-over effect 
(one group higher than the other at one end of the pretest score range and 
the opposite group higher at the other end; see Figure 2), the t-test 
made at the combined pretest mean would likely indicate a small, insignifi- 
cant difference between groups. Using a model without the interaction term 
to analyze the same data would also likely yield a small, insignificant 
difference between groups over the entire pretest score range because the 
actual changing group differences over this range would tend to cancel 
each other out. 

The Method Employed for Obtaining Results With the Extended 
Variables Model , In order to assess the impact of the experimental program 
for any grade/site/subject, the group coefficient, is tested for 
significance using a t-test of the null hypothesis! B^ * 0, at the 0,05 
level of significance. Usually this means values of t greater than 2,0 
are taken as evidence for a significant group difference. In some cases 
a value of t greater than 2,0 is required as the critical value, depending 
on the sample size and the number of variables in the model. 
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Suimnary 



In summary, the analysis method used is regression analysis. 
Regression models are fitted at all grade/site combinations for both reading 
and mathematics. The Pre-Post Model includes pretest scores , group member- 
ship, and a group-pretest interaction as independent variables. The Extended 
Variables Model includes pretest scores, group membership, total family 
income, race, father’s education, and an attitudinal item assessing attitudes 
of parents toward their children being taught by a new teaching method. The 
Pre-Post Model is also applied in additional analyses comparing the experi- 
mental group with the comparison group and the special treatment groups, com- 
paring the experimental and control groups on attendance, and comparing 
the experimental and control groups on second year test scores obtained 
at the beginning of the 1971-72 school year. All of these applications are 
at only certain grade/sites. These additional analyses are discussed in 
greater detail in the following section on results. 
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RESULTS AND CONCLUSIONS- 
TECHNOLOGY COMPANY SITES 



Introduction 



The purpose of this section is to present the principal results 
obtained from a variety of analyses performed on the data. These results 
are presented under several headings. The main analyses are presented first. 
In general, the main analyses consist of comparing experimental and control 
groups at each site/grade/subject combination, utilizing in regression models 
pre- and post achievement test scores and data collected on other variables 
during the experimental (1970-71) year. The first part of these main 
analyses is the application of the Pre-Post Model, where E/C comparisons 
are made by regressing posttest scores against pretest scores. The resulting 
regression lines are then evaluated at the pretest mean of the combined 
experimental and control groups. The difference between these regression 
lines are tested for statistical significance using a formal t-value equal 
to 2.0. These results constitute the primary results of the Pre-Post Model 
analyses, and are summarized in Table 27. 

The second part of the main analyses consists of applying, again 
at each site/grade/subject combination, extended regression models which 
include family background and attitudinal variables in addition to pretest. 
Results from these extended regression models are summarized in Table 30, 

The additional variables included in these extended models are family income, 
father's education, race of student, and one attitudinal item from the 
Parent Questionnaire, The attitudinal item included assesses parents' 
attitudes toward their children being taught by a new teaching method. As 
discussed earlier in the Target Population Section, E/C group differences 
existed very frequently on this item, and for this reason the item was 
included as a variable in the extended regression models. 

The main analyses at each site are followed by an aggregate 
analysis across sites. This analysis is descriptive in nature, not involving 
any regression or other formal statistical models, and consists of an 
examination of pre- and posttest means for the experimental and control 
groups at each grade and in each subject area, aggregated across sites. 
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The changes in the grade equivalents of these means are then discussed. 

In the next part of this section, results from analyses 
concerned with assesing the stability of the treatment impact are presented. 
To make this assessment, experimental and control students from 46 selected 
site/grade/subject combinations were administered the evaluation test during 
the fall of the school year that followed the experimental year. Those 
grade/site/subject combinations showing positive (or negative) impacts in 
the spring are examined to determine whether the impact remained positive 
(or negative) into the following year. These results are discussed and 
then summarized in Table 34, Because the original samples of both the 
experimental and control students suffered normal attrition during the 
summer months the pretest means for the reduced samples were computed 
and compared with those of the original samples, A discussion of these 
findings concludes the assessment of the stability of treatment impact. 

As part of the experimental design for this program, many sites 
had "replacement" groups. These groups of students were generally taught 
in the same building as the treatment students. They were pretested and 
posttested as though they were control students. In some instances, as 
intended, replacement students were transferred out of their replacement 
groups and became members of experimental treatment groups whenever students 
were lost from the experimental treatment groups because of dropouts, etc. 
Where such transfers were few, the replacement groups were not severely 
.depleted and became eligible for consideration as a kind of "in-house" 
control group (referred to in the analysis as "comparison" groups) , This 
is in contrast to the normal control groups that were taught in other 
buildings. The posttest performance of students in 58 comparison groups 
are analyzed using the Pre-Post Model, Two regression analyses involving 
comparison groups are made. The first of these consists of comparing the 
posttest performance of students in the comparison groups with that of the 
control groups;, the second consists of comparing the posttest performance 
of students in the comparison groups with that of the experimental groups. 
The results of these analyses are suranarized in Tables 37 and 38 , Finally, 
these two regression analyses are interrelated as shown in Table 39, 



89 



86 



In addition to the experimental groups under the guidance of the 
technology companies, the experimental design also included some "special 
treatment" groups at Grand Rapids, Michigan, and Hartford, Connecticut, 
These groups were part of other remedial education programs or projects 
in these two school systems. The posttest performance of students in 
these special treatment groups is compared with experimental and control 
groups at these sites. The results of these analyses are given in Tables 

41 through 45, 

Because of the possible importance of student attendance in 
the assessment of program impact, a special analysis of attendance was 
made which compared the experimental and control groups. This analysis 
is summarized in Table 47 » 

The section concludes with a sunanary of results and conclusions. 

The Main Analyses: Regression Analyses of 

Experimental verses Control Groups on 
First Year Test Results 



The purpose of this section is to present the principal results 
based upon tests of significance of the difference between the experi- 
mental and control groups’ estimated posttest means, using the two types 
of regression model si the Pre-Post Model and the Extended Variables Model, 

It is important to emphasize that results and comparisons are given in terms 
of estimated posttest means and not the actual posttest means. Estimated 
posttest means are generated from the regression models fit to the data. 

As discussed previously in the Method of Analysis section, any comparison 
of actual posttest means suffers from the fact that 1 and G groups differed 
on the pretest, so that any group differences between actual means on the 
posttest can be partly attributed to differences between the groups in their 
initial entry level, as well as partly attributed to differences between the 
groups on other variables, such as family background variables. The 
regression analysis technique copes with this problem by providing 
estimated posttest means for the 1 and C groups at a given common pretest 
value, so that any difference in the E and C estimated posttest means cannot be 
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attributed to original differences between the groups on the pretest. 

For the Extended Variables Model, where group differences . in other variables 
in addition to pretest are taken into account, this concept is extended 
so as to provide estimated means and differences between estimated means 
"holding constant" or "partial ing out" the variables on which the groups 
might differ. 

However, actual posttest means are provided in Appendix K for 
the interested reader, for experimental and control groups, for each site 
in each grade and subject area. Corresponding grade equivalent values 

are also given (shown in parentheses beside the raw score mean). Also, 

! ! 

pretest raw score means and associated grade equivalents are given in 
Appendix K, along with standard deviations, sample sizes, and pre-post 
correlation coefficients. 

The students in the regression analyses given below are full-year 
students, i.e. , those identified as being in either the experimental or 
control group from a time within 3 weeks of the administration of the 
pretest to the time of the posttest administration. Further, each student 
was required to have both a pre- and post test score in order to be included 
in the analysis. The numbers of such students in the E and C groups are 
given in Appendix N.* 

% 

Pre-Post Model Results 



Pre-Post Model results are presented below for reading and for 
mathematics, within each grade. Tables and descriptive discussions show 
the sites for which statistically significant group differences were found, 
the estimated posttest means for each group, their differences, and the 
combined E and C pretest mean associated withthe posttest differences,** 
All analyses are done with raw scores j however, the grade 
equivalents of the raw scores are presented also to facilitate the inter- 




* Appendix N also provides the number of students in the comparison and 
special treatment groups analyses, 

** Appendix R presents these data for all grade/site/subject combinations 
regardless of whether a statistically significant difference was or was 
not found. Also, the associated t-ratios are given in this appendix, 
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pretation of the results through the use of a commonly used metric. 

Table 21 shows Grade 1 results for reading and mathematics 
for those sites exhibiting a statistically significant group difference 
at the combined E and C pretest raw score mean,** The sites are shown 
in Column 1. Column 2 shows the stanine value eorresponging to the 
combined E and C pretest raw score mean. The estimated posttest means 
shown in Columns 3 and 4 were obtained by evaluating the fitted E and 
C regression lines at the mean pretest value. The differences between 
these estimated posttest means are shown in the last column. The numbers 
in parentheses in the last three columns give the grade equivalents that 
correspond to the mean raw scores and their differences. 

The table shows, for example, that the difference between the 
estimated raw score posttest means at Selmer is +13 and this difference 
corresponds to a grade equivalent difference of +0,7, This raw score 
difference is equal to the difference between 68 for the experimental 
posttest mean and 55 for the control posttest mean, as estimated by evaluating 
the fitted regression lines at the mean pretest value. 

An examination of. the results in this table shows that the largest 
positive differences in posttest raw score means for reading and mathematics 
are equal to 21 and 14, respectively, and both of these occur at Jacksonville, 
The largest positive differences in grade equivalents occur in reading at 
Selmer (+0,7) and in mathematics at Jacksonville (>+0,6), The largest 
negative difference in raw score posttest means occurs in reading at 
Wichita (-16), In terms of grade equivalents, the largest negative 
difference also occurs in reading at Wichita (-1,0), Overall, positive 
differences in posttest means occur in two sites in reading and in three 
sites in mathematics. 



** Since Battelle's Interim report, the Grade 1 results have been re-analyzed 
in that scores of a zero posttest have been dropped from the analysis. 
Results from this analysis are in this report, and are taken to be 
definitive. The dropping of students with zero posttest scores resulted 
in three fewer impacts in favor of the E group than reported in the 
Interim report, and six more impacts in favor of the control group, in 
reading and mathematics combined. 
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TABLE 21 9 ESTIMATED EXPERIMENTAL (E) AND CONTROL (C) POSTTEST MEANS 
IN READING AND MATHEMATICS AND DIFFERENCES BETWEEN THESE 
mms FOR SITES EXHIBITING A SIGNIFICANT GROUP DIFFERENCE 
IN GRADE 1 



Site 


Mean 

Pretest 

Value* 


Estimated 
Post test Means 


E/C 

Difference 


E 


C 


Selmer 


(3) 


68 (1.6) 


55 ( .7) 


+13 (+.7) 


Wichita 


(3) 


53 ( .6) 


69 (1,6) 


-16 (-1.0) 


Las Vegas 


(2) 


51 «. 6) 


57 ( .8) 


-6 «-.2) 


Philadelphia 


(2) 


51 «, 6) 


57 < ,8) 


-6 (<“. 2) 


Seattle 


(4) 


63 (1.2) 


73 (1,8) 


-10 (- .6) 


Portland 


(4) 


62 (1.2) 


71 (1.7) 


-9 (-.5) 


Jacksonville 


(2) 


57 ( .8) 


36 (<,6) 


+21 (>+, 2) 


Hammond 


(3) 


62 (1.2) 


73 (1,8) 


-11 (-.6) 



MATH 



Site 


Mean 

Pretest 

Value* 


Estimated 
Posttest Means 


E/C 


E 


C 


Difference 


Dallas 


(1) 


34 (1,0) 


29 ( .6) 


+5 (+.4) 


Las Vegas 


(2) 


27 «,6) 


40 (1,3) 


-13 «-. 7) 


Fresno 


(4) 


33 ( .9) 


42 (1.5) 


-9 (-.61 


Philadelphia 


(2) 


28 «. 6) 


38 (1,2) 


-10 «-,6) 


Grand Rapids 


(2) 


32 ( ,9) 


41 (1.4) 


-9 (-.5) 


Hartford 


(2) 


30 ( .7) 


39 (1.3) 


••9 (-.6) 


McComb 


(2) 


44 (1,6) 


36 (1.1) 


+8 (+.5) 


Portland 


(4) 


41 (1.4) 


55 (2.0) 


-14 (-.6) 


Jacksonville 


(2) 


38 (1,2) 


24 «. 6) 


+14 (>f,6) 



* The stanino value corresponding to the combined E and C pretest 
raw score mean. 



f 
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Table 22 gives the results for Grade 2 for those sites showing 
a significant group difference at the combined raw score mean for the 
E and C groups. The mean pretest values in Column 2 are given in raw 
score units along with corresponding grade equivalents. The regression 
estimates of the posttest means and the differences between them for the 
E and C groups are given in Columns 3, 4, and 5, Raw score values and 
corresponding grade equivalents are given for the estimated posttest 
means , 

The largest positive differences in the posttest means occur 
at Jacksonville (+6) for reading and at Dallas (+9) for mathematics. The 
largest negative differences in these means occur at Las Vegas for both 
reading (-14) and mathematics (-7), In reading the grade equivalent dif- 
ferences range between a maximum at Wichita (0*2) and a minimum at Las 
Vegas or Bronx (-0.3) ; in mathematics this range is associated with 
Jacksonville (+0,4) and Rockland (-0.5) . The E group at Rockland in 
mathematics shows a larger grade equivalent gain than any other I group 
in the table (2, 6-1. 7 ss 0, 9) , All posttest means have grade equivalents 
less than 2,9 ("average" grade-level position at the end of the school 
year is 3,0) with the exception of the control group at Rockland in 
mathematics (3,1), The table shows 3 positive and 7 negative differences 
in posttest means in reading; and 2 positive and 7 negative differences 
in mathematics. 

Table 23 shows the regression results for Grade 3, The maximum 
and minimum differences in raw score units are at Selmer (+9) and Seattle 
(-11) for reading, Dallas (+20) and either Las Vegas or Hartford (-13) for 
mathematics, Selmer and Seattle also yield the maximum and minimum dif- 
ferences in grade equivalents for reading, (0.4) and (-0,5), respectively. 

In mathematics these extremes are associated with Dallas (+0.6) and either 
Hartford or MeComb (-0,4), It should be noted that a gain of 1,3 grade 
equivalents occurred for the E group in mathematics in Selmer, The results 
show 3 positive and 4 negative differences in the posttest means for reading, 
and 5 positive and 8 negative differences in the posttest means for math- 
ematics. 
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TABLE 22. ESTIMATED EXPERIMENTAL (E) AND CONTROL (C) POSTTEST MEANS 
IN READING AND MATHEMATICS AND DIFFERENCES BETWEEN THESE 
MEANS FOR SITES EXHIBITING A SIGNIFICANT GROUP DIFFERENCE 
IN GRADE 2 



READING 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E 


C 


Selmer 


40 (1,7) 


65 (2,2) 


60 (2.1) 


+5 (+.1) 


Wichita 


39 (1.7) 


63 (2,2) 


58 (2.0) 


+5 (+.2) 


Las Vegas 


23 (1,3) 


32 (1.5) 


46 (1,8) 


-14 (-.3) 


Taft 


37 (1,6) 


52 (1,9) 


57 (2,0) 


-5 (-.1) 


Hartford 


27 (1.4) 


42 (1.7) 


51 (1.9) 


-9 (-.2) 


Me Comb 


29 (1.4) 


47 (1.8) 


55 (2,0) 


-8 (-.2) 


Seattle 


39 (-1.7) 


58 (2.0) 


65 (2.2) 


-7 (-.2) 


Portland 


42 (1.8) 


59 (2,1) 


65 (2,2) 


-6 (-.1) 


Jacksonville 


27 (1.4) 


50 (1,9) 


44 (1,8) 


+6 (+.1) 


Bronx 


28 (1.4) 


41 (1.7) 


54 (2,0) 


-13 (-.3) 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E . 


C 


Dallas 


21 (1.2) 


34 (1.5) 


25 (1.3) 


+9 (+.2) 


Rockland 


38 (1.7) 


53 (2.6) 


56 (3,1) 


-3. (-.5) 


Las Vegas 


20 (1.2) 


29 (1.4) 


36 (1,6) 


-7 (-.2) 


Fresno 


31 (1,4) 


43 (2.0) 


47 (2.2) 


-4 (-.2) 


Philadelphia 


23 (1,3) 


36 (1.6) 


40 (1,8) 


-4 (-.2) 


Seattle 


32 (1.5) 


45 (2,1) 


51 (2,4) 


-6 (-.3) 


Portland 


39 (1,7) 


49 (2.3) 


52 (2,5) 


-3 (-.2) 


Jacksonville 


23 (1.2) 


44 (2,0) 


36 (1,6) 


+8 (+.4) 


Bronx. 


22 (1.2) 


40 (1.8) 


43 (2,0) 


-3 (-.2) 



0 
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TABLE 23 * ESTIMATED EXPERIMENTAL (E) AND CONTROL (C) POSTTEST IffiANS 
IN READING AND MATHEMATICS AND DIFFERENCES BETWEEN THESE 
MEANS FOR SITES EXHIBITING A SIGNIFICANT GROUP DIFFERENCE 
IN GRADE 3 



Site 


Mean 

Pretest 

Value 


Estimated 
Post test Means 


E/C 

Difference 


E 


C 


Selmer 


45 (2,4) 


67 (3.2) 


58 (2.8) 


+9 (+.4) 


Dallas 


26 (1.8) 


36 (2.2) 


29 (1,9) 


+7 (+.3) 


Las Vegas 


28 (1.9) 


39 (2.3) 


47 (2.4) 


-8 (-.1) 


Seattle 


46 (2.4) 


53 (2.6) 


64 (3.1) 


-11 (-.5) 


Jacksonville 


30 (2.0) 


44 (2.4) 


38 (2.2) 


+6 (+#2) 


Hammond 


42 (2.3) 


54 (2.7) 


59 (2.8) 


-5 (-.1) 


Bronx 


31 (2,0) 


47 (2,4) 


53 (2.6) 


-6 (-.2) 



MATH 



Site 


Mean 

pretest 

Value 


Estimated 
Post test Means 


E/C 

Difference 


E 


C 


Selmer 


59 (2,4) 


88 (3.7) 


75 (3.2) 


+13 (+.5) 


Athens 


50 (2.3) 


68 (2,9) 


63 (2.6) 


+5 (+.3) 


Wichita 


39 (1.9) 


53 (2.3) 


60 (2.5) 


-7 (-.2) 


Dallas 


35 (1,6) 


59 (2,5) 


39 (1.9) 


420 (+.6) 


Las Vegas 


37 (1,7) 


49 (2,3) 


62 (2,6) 


-13 (-.3) 


Fresno 


51 (2.3) 


62 (2.6) 


68 (2,9) 


-6 (-.3) 


Taft 


52 (2,3) 


71 (3.0) 


65 (2,7) 


+6 (+.3) 


Hartford 


44 (2.2) 


51 (2,3) 


64 (2,7) 


-13 (-.4) 


McComb 


44 (2.2) 


62 (2,6) 


70 (3,0) 


-8 (-.4) 


Seattle 


. 53 (2,3) 


71 (3.0) 


78 (3.2) 


-7 (-.2) 


Portland 


58 (2,4) 


74 (3,1)' 


80 (3,3) 


-6 (-.2) 


Jacksonville 


38 (1.8) 


58 (2,4) 


51 (2,3) 


+7 (+.1) 


Harmnond 


53 (2,3) 


67 (2,8) 


71 (3.0) 


-4 (-.2) 
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Table 24 gives the results from the regression analyses for Grade 
7, Only 2 sites in reading, Athens and Rockland, and 4 sites in mathematics , 
Rockland, Fresno, Grand Rapids, and Seattle, yielded significant group dif- 
ferences, The only positive difference between the post test means occurred 
at Athens in reading. The table also shows that the larger grade equivalent 
gains were in two C groups in mathematics (1.2 for Rockland and 1,3 for 
Seattle), In reading there was one positive and one negative difference in 
posttest means. In mathematics all 4 differences were negative. 

Table 25 shows the results obtained for Grade 8, In reading, 
the largest positive difference between the regression estimates of the 
posttest means occurred at Grand Rapids (+7) ; the largest negative dif- 
ference occurred at Seattle (-10), In mathematics, there are no positive 
differences and the largest negative difference occurs at Seattle (-20), 

A large grade equivalent gain of 1,4 occurred for the E groups in reading 
at Anchorage and Grand Rapids; similar gains of 1,3 occurred for the C 
groups in Rockland, Seattle, and Hammond. A negative gain (-0,6) in grade 
equivalents occurred for reading in the E group of Philadelphia, In 
mathematics, the table shows that the grade equivalent gain for the G 
group at every site exceeds 1,0 with the exception of Hammond (0,9), The 
largest negative difference occurred for Seattle, where the E group showed 
no gain in grade equivalents and the C group showed a gain of 1,6 in grade 
equivalents. Three sites showed positive impacts in reading; no sites 
showed positive impacts in mathematics. 

Table 26 shows the results obtained for Grade 9, In both reading 
and mathematics, 3 sites showed positive differences in the regression 
estimates of the posttest means. In reading, the largest gains in grade 
equivalents were made by the 1 group at Athens (1,2) and by the G group 
at Seattle (1,1). In mathematics, both the minimum and maximum gains in 
grade equivalents occurred at Seattle where the E group showed a grade 
equivalent gain of 0,6 and the C group showed a grade equivalent gain of 
1,5. 
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TABLE 24. ESTIMATED EXPERIMENTAL (E) AND CONTROL (C) POSTTEST MEANS IN 
READING AND MATHEMATICS AND DIFFERENCES BETWEEN THESE MEANS 
FOR SITES EXHIBITING A SIGNIFICANT GROUP DIFFERENCE IN 
GRADE 7 



READING 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E 


C 


Athens 


38 (4.4) 


43 (4.8) 


39 (4.5) 


+4 (+.3) 


Rockland 


56 (5.8) 


59 (6.0) 


64 (6.6) 


-5 (-.6) 



Site 


Mean 

Pretest 

Value 


Estimated 
Post test Means 


E/C 

Difference 


E 


C 


Rockland 


62 (5.7) 


72 (6,4) 


79 (6.9) 


-7 (-.5) 


Fresno 


36 (4.3) 


42 (4,6) 


52 (5,2) 


-10 (-.6) 


Grand Rapids 


38 (4.4) 


44 (4.7) 


50 (5.1) 


-6 (-.4) 


Seattle 


57 (5,4) 


59 (5,6) 


75 (6.7) 


-16 (-1.1) 
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TABLE 25. ESTIMATED EXPERIMENTAL (E) AND CONTROL (C) POSTTEST MEANS 
IN READING AND MATHEMATICS AND DIFFERENCES BETWEEN THESE 
MEANS FOR SITES EXHIBITING A SIGNIFICANT GROUP DIFFERENCE 
IN GRADE 8 



READING 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E 


C 


Dallas 


23 (3,7) 


31 (4.6) 


27 (4.2) 


+4 (+. 4 ) 


Anchorage 


46 (6,6) 


59 (8.0) 


53 (7,3) 


+6 (+.7) 


Rockland 


47 (6.7) 


55 (7.4) 


59 (8.0) 


-4 (-.6) 


Fresno 


30 (4,5) 


33 (4.9) 


38 (5,6) 


-5 (-.7) 


Philadelphia 


26 (4,1) 


22 (3.5) 


31 (4,6) 


-9 (-1.1) 


Grand Rapids 


32 (4,8) 


42 (6.2) 


35 (5,2) 


+7 (+1,0) 


Seattle 


41 (6,0) 


43 (6,2) 


53 (7,3) 


-10 (-1,1) 


Hammond 


40 (5.8) 


47 (6,7) 


51 (7.1) 


-4 (-.4) 



MATH 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E 


C 


Rockland 


56 (6,8) 


66 (7.8) 


70 (8.1) 


-4 (-.3) 


Fresno 


34 (4.8) 


37 (5.4) 


42 (5.9) 


-5 (-.5) 


Grand Rapids 


38 (5,3) 


39 (5.7) 


47 (6.4) 


-8 (-.7) 


Seattle 


47 (6.0) 


44 (6.0) 


64 (7,6) 


-20 (-1,6) 


Portland 


47 (6.0) 


52 (6.7) 


60 (7.3) 


-8 (-.6) 


Hammond 


46 (6.0) 


51 (6.7) 


55 (6,9) 


-4 (-.2) 
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TABLE 26 # ESTIMATED EXPERIMENTAL (E) AND CONTROL (C) POSTTEST MEANS 

IN READING AND MATHEMATICS AND DIFFERENCES BETWEEN THESE MEANS 
FOR SITES EXHIBITING A SIGNIFICANT GROUP DIFFERENCE IN GRADE 9 



READING 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E 


C 


Athens 


34 (5.0) 


43 (6.2) 


39 (5.7) 


+4 (+.5) 


Las Vegas 


49 (6.9) 


58 (8,0) 


52 (7,3) 


+6 (+, 7) 


Philadelphia 


23 (3.7) 


24 (3,8) 


27 (4,2) 


-3 (-.4) 


Taft 


36 (5.3) 


45 (6,4) 


37 (5.5) 


+8 (+.9) 


Seattle 


49 (6.9) 


54 (7,4) 


58 (8,0) 


-4 (-,6) 


Portland 


47 (6.7) 


53 (7,3) 


56 (7,6) 


-3 (-.3) 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E 


C 


Selmer 


59 (6.9) 


68 (8,0) 


73 (8.4) 


-5 (-.4) 


Athens 


41 (5.6) 


53 (6,8) 


.48 (6,4) 


+5 (+.4) 


Anchorage 


53 (6.6) 


65 (7.8) 


55 (6,9) 


+10 (+.9) 


Rockland 


70 (7.8) 


78 (9.0) 


70 (8,1) 


+8 (+.9) 


Seattle 


54 (6,7) 


60 (7,3) 


71 (8,2) 


-11 (-.9) 


Portland 


54 (6,7) 


61 (7,5) 


67 (8,0) 


-6 (-.5) 
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Summary of Pre-Post Model Results 

Table 27 shows a summary of the results obtained with the pre- 
post regression model for each grade, site, and subject. The symbols, E 
and C, indicate the occurrence of a significant positive or negative group 
difference, respectively, between the posttest means as estimated by the 
regression lines fitted to the pre-post data. The marginal row and colinnn 
totals show the number of E's and C's together with the number of empty 
cells where a nonsignificant (Not Sig.) group difference was obtained. 

The table shows, for example, in Grade 2 -reading, the positive 
impacts (E) occurred at Selmer, Wichita, and Jacksonville; the negative 
impacts (C) occurred at Las Vegas, Taft, Hartford, MeComb, Seattle, 

Portland, and Bronx. The marginal totals for this column show 3 positive 
impacts; 7 negative impacts; and 8 sites where the group differences were 
not significant. An examination of the row for Wichita shows a negative 
impact in Grade 1 -reading; a positive impact in Grade 2 -reading; a negative 
impact in Grade 3-mathematics; and no significant impact in any other grade/ 
subject combination. The marginal totals for Wichita show one positive 
impact, 2 negative impacts; and 9 instances where the impacts were not 
s ignif icant , 

An examination of the column totals shows that five positive 
impacts were obtained for Grade 3 in mathematics. No other grade/subject 
combination shows a greater number of positive impacts. The smallest 
number of positive impacts for grade/subject combinations occurred for 
Grades 7 and 8 in mathematics where no positive impacts were obtained. 

The corresponding maximum and minimum number of negative impacts are seen 
to be 8 for Grade 3-mathematics, and one for Grade 7-reading. 

In none of the 12 grade/subject combinations did the number of 
positive impacts exceed the ntmber of negative impacts. 

An examination of the row totals shows that Jacksonville 
exhibited six positive impacts. The seven sites showing no positive 
impacts are Fresno, Philadelphia, Hartford, Seattle, Portland, Hammond, 
and Bronx. The maximisn nimber of negative impacts .is shown by the ten 
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negative impacts at Seattle, No negative impacts occurred at Athens* 

Dallas, Anchorage, and Jacksonville. The difference between the number 
of positive impacts and the number of negative impacts is seen to be a 
maximum of 6 for Jacksonville and a minimum of -10 for Seattle, 

The lower right-hand totals show that 28 positive impacts 
occurred, as compared to 60 negative impacts, Thu s , the control groups 
exhibited statistically superior performance as determined by the regression 
analyses in approximately twice or many instances or did the experimental 
groups. However, by far the major outcome was no statistically significant 
difference between the groups, since 124 differences ware not significant. 

Of the 28 positive impacts, almost twice as many (18) occurred 
at the Elementary level than at the Secondary level (10), However, at 
either level the percentage of positive impacts out of all grade/site/ 
subject combinations is very low, being 17 and 9 percent respectively. 

It is also interesting to note that 23 of the 28 positive impacts occurred 
at sites associated with three of the six technology companies, as shown 
in Table 28. However, such a comparison needs to be interpreted with 
caution, since different companies did not conduct programs in the same 
sites. Therefore, it is impossible to disentangle the site from the company 



Extended Variables Model Res ul t s 



The purpose of this section is to present the results of compari- 
sons between the posttest performance of the experimental (E) and control 
(G) groups taking into account group differences on selected variables in 
addition to the pretest score. These variables are taken into account 
quantitatively through inclusion in the extended regression model described 
earlier. In review, this model is an extension of the Pre-Post Model without 
the interaction term between pretest and group. The additional variables 
selected were student race, father's education, total family income, and 
level of parents' approval of new instructional methods for their children 
(Item No. 10 on the Parent Question and referred to as "Approve New Method"), 
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TABLE 2 



O 

ERIC 



8 , NUMBER OF POSITIVE BffACTS FROM THE PRE-POST MODEL ANALYSES 

FOR THE ELEMENTARY AND JUNIOR HIGH GRADES, FOR EACH TECHNOLOGY 
COMPANY 



Company 


Elementary 


Secondary 


Totals 


A 


6 


3 


9 


B 


4 


4 


8 


C 


0 


1 


1 


D 


1 


2 


3 


I 


1 


0 


1 


F 


6 


0 


6 



Totals: 18 10 28 



A: Sclmer, Athens, Wichita 

B: Dallas, Anchorage, Rockland 

Ct Las Vegas, Fresno, Philadelphia 

D: Taft, Grand Rapids, Hartford 

E: McComb, Seattle, Portland 

F: Jacksonville, Hammond, Bronx 
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The rationale for selecting these variables and dropping the 
group-by-pretest interaction is discussed in the previous section on the 
methods of analysis. The results of the Extended Variables Model are 
presented here. 

There are certain disadvantages in using the Extended Variables 
Model to assess program impact, A smaller data base is available for 
this model than with the Pre-Post Model because it contains only students 
whose parents returned a questionnaire. Not only are the samples smaller 
in size (generally 40-50 percent as large as the samples in the Pre-Post 
Model analyses) , but also the samples could be different in the type of 
students in them. However, a comparison of combined E and C pretest means 
of the samples associated with the Pre-Post and Extended Variables Models 
showed that these pretest means were close, most often not differing by 
more than one or two raw score units. There were, however, exceptions, 
particularly at Hartford and Grand Rapids, where the corresponding pretest 
means often differed on the order of four to six raw score units. In 
general, when the means of the samples associated with the two models din 
differ, the sample associated with the Extended Variables Model (EVM) had a 
higher pretest mean. 

Although there are the above disadvantages, by use of this 
model group differences on variables in addition to pretest can be 
taken into account. Thus, in this sense, the results of these extended 
model analyses do provide a better comparison of experiments, a* d control 
groups. 

Table 29 describes the additional variables included in the 
extended variables model for each grade/site. Note that no EVM was fitted 
at Rockland , Philadelphia, and Grades 1, 7, 8, and 9 at Bronx, 
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TABLE 29, INDEPENDENT VARIABLES CONSTITUTING TEE EXTENDED 
VARIABLES MODEL FOR EACH GRADE/SITE COMBINATION * 



Site 


Grade 


Group 


Independent Variaj 

Student * s 
Pretest Race 


hies 

Father's 

Education 


Total Family 
Income 


Approve 
New Method 


Selmer 


All 


X 


X 




X 


X 


X 


Athens 


All 


X 


X 


X 


X 


X 


X 


Wichita 


/ All 


X 


X 


X 


X 


X 


X 


Dallas 


Ail 


X 


X 




X 


X 


X 


Anchorage 


All 


X 


X 


X 


X 


X 


X 


Rockland ** 


None 














Las Vegas 


1 


X 


X 


X 








Las Vegas 


2-9 


X 


X 


X 


X 


.X 


X 


Fresno 


All 


X 


X 


X 


X 


X 


X 


Philadelphia ** 


None 




' 










Taft 


All 


X 


X 




X 


X 


X 


Grand Rapids 


All 


X 


X 


X 


X 


X 


X 


Hartford 


All 


X 


T f 

A 




X 


X 


X 


Me Comb 


All 


X 


X 


X 


X 


X 


X 


Seattle 


Ail 


X 


X 


X 


X 


X 


X 


Portland 


All 


X 


X 




X 


X 


X 


Jacksonville 


All 


X 


X 




' X 


X 


X 


Haranond 


All 


X 


X 


X 


X 


X 


X 


Bronx ** 


1&9 














Bronx 


2&3 


X 


X 


X 


X 


X 


X 


Bronx **■ 


7&8 






- 









* Presence of variable in EVM is indicated by an "X" • 
** No EVM's fitted due to insufficient or no data. 
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Table 30 presents the results of the Extended Variables Models (EVM) 
along with the results from the Pre-Post Model (PPM) s so that comparisons of 
results from the two models can be made. For each grade/site/subject the 
following entry is made* "E" if the difference between groups was statis- 

i * 

tieally significant in favor of the E group, a "G" if the difference was in 
favor of the C group, and a blank if the difference was not significant. 
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TABLE ' 30 . COMPARISON OF SIGNIFICANT E/C GROUP DIFFERENCES FOUND 1 USING THE, PRE-POST MODEL (PPM) 
VERSUS THE: EXTENDED VARIABLES MODEL (EVM) FOR EACH GRADE, SITE, AND SUBJECT 
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The column totals give, the number of "E's" and "C's" and blanks (Not. Sig.) 
for each grade/subject, summed across sites. For example, in Grade 8- 
Reading, of the 15 sites where both the EVM and PPM were employed, there 

were three "E's" and three "C's" as a result of using the PPM, but no 

''E's 11 and only one "C" based upon the EVM. ■ This type of result--a loss in 
number of both "E's" and "C's"--is by far the most typical. Note, however, 
that in Grade 7-Mathematics there were no ’’E's" and three "C’s" using the 
PPM, but two "E's" and only or 1 "C" using the EVM. 

The row totals show the number of "E's", "C's" and blanks for 
each site summed across grades and subjects for those sites where an 

Extended Variables Model was used. None of the sites gained in the number 

of "E’s" or "C’s" (with the exception of McComb, which gained one 1). 

Of the four sites which had a relatively large number of "E’s" using the 
PPM (Selraer, Athens, Dallas, and Jacksonville), only Se liner maintained 
the same number using the EVM, with Athens, Dallas, and Jacksonville 
losing in the number of "E's", In fact, of the 10 sites exhibiting at 
least one positive impact with the Pre-Post M odel and where the Extended 
Variables Model was employed, in 8 of the 10 sites the number of positive 
impacts was reduced when using the EVM, 

The totals in the lower right comer of the table describe 
the overall result. Where 26 "E's" were obtained with the PPM, only 15 
were obtained with the EVM, in those grades and sites where both models 
were used. Forty-nine "C's" were found using the PPM, only 27 using the 
EVM. The corresponding number of nonsignificant outcomes went from 109 
with the PHI to 142 with the EVM. The number of "E's" lost, 11, represents 
42 percent of the number obtained with the PPM. The number of "C's", 22, 
represents 45 percent. Thus, the change to the EVM does not appear to have 

favored either group. 

Although the change to the Extended Variables Model does not 
favor either group, the conclusion is nonetheless indicated that application 

of the Extended Variables Model provides even less evidence in favor of an 
experimental group impact than the Pre-Post Model, simply because the pro- 
portion of positive impacts is less. And, in general, those few sites 
exhibiting a relatively large number of positive impacts with the Pre-Post 
Model lost some of these positive impacts when the extended model wan applied. 
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Experimental - Control Group Comparisons Aggregated Across Sites 

Although the primary emphasis of the analyses of data in this 
report is on obtaining results at a grade/site level, by means of regression 
analyses, it is informative to make descriptive comp isons of the experimen 
tal and control gourps' average performance on the posttest aggregated over 
the 18. sites. Table 31 shows the raw score means (and associated grade 
equivalents)* of the pre- and posttests for the E and C groups at each 
grade for all 18 sites combined. As shown there, the differences in 
pre-post gains between the two groups, either in raw scores or grade 
equivalents, are very small. In terms of raw score units, the largest 
difference in gain between the two groups occurs in Grade 3 reading, 
where the experimental group gained 16 raw score points, as compared to 
a gain of 13 raw score points for the control group. However, in terms 
of grade equivalent units for Grade 3 reading, there is no difference in 
gains between E and C groups, each group gaining 0,4 grade equivalents. 

The largest difference in gains between the two groups in terms 
of grade equivalent units occurs at Grade 3 mathematics, where the experi- 
mental group gained 0.4 grade equivalents, as compared to a gain of 0,7 
grade equivalents for the control group. Note, however, that the difference 
in gain between the groups in raw score units is zero, each group in 
Grade 3 mathematics gaining 18 raw score units. 

Figure 3 portrays the data in Table 31 graphically, by showing 
a plot of grade equivalent gains as a function of grade level. (No results 
are shown for Grade 1 because the pretest did not yield grade equivalents,) 
For mathematics, the upper plot shows, for example, that the third grade 
experimental group began the third grade with an average grade equivalent 
of 2,2, and ended the third grade with an average grade equivalent of 2,6, 

* The grade equivalents were obtained by first calculating the raw score 
mean, and then detarmining the grade equivalent corresponding to this 
mean raw score value in the publisher's grade equivalence tables. 
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FIGURE 3, GRADE EQUIVALENT GAINS BY GRADE FOR MATHEMATICS 
AND READING 
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TABLE 31. OVERALL MEAN PRETEST AND POSTTEST VALUES (AND ASSOCIATED GRADE 
EQUIVALENTS) FOR EXPERIMENTAL (E) AND CONTROL ,(C) STUDENTS, BY 
SUBJECT AREA AND GRADE 



MATHEMATICS 





Grac 

E 


ie 1 
C 


Grac 

E 


ie 2 
C 


j 

Grac 

E 


!e 3 
• C 


Grac 

E 


ie 7 
C 


Grac 

E 


ie 8 
C 


Grac 

E 


ie 9 
C 


Pretest 


68 

(2)* 


74 

(3)* 


28 

(1.4) 


31 

(1.4) 


44 

(2,2) 


51 

(2.3) 


43 

(4.7) 


49 

(5.1) 


39 

(5.4) 


45 

(5.9) 


46 

(6.0) 


53 

(6.6) 


Posttest 


38 

(1.2) 


43 

(1.5) 


42 

(1.9) 


45 

(2.1) 


62 

(2.6) 


69 

(3.0) 


53 

(5.3) 


60 

(5.7) 


45 

(6.2) 


53 

(6.8) 


53 

(6.8) 


60 

(7.3) 


Raw Score 
Gain 


- 


- 


14 


14 


18 


18 


10 


11 


6 


8 


7 


7 


GE Gain 


- 


- 


0.5 


0.7 


0.4 


0.7 


0.6 


0.6 


0.8 


0.9 


0.8 


0.7 



Sample 1042 1082 1152 1106 1242 1115 1197 1089 1065 1088 1102 975 

Size 



READING 





Gra< 

E 


ie 1 
C 


Grac 

E 


ie 2 
C 


Grac 

E 


ie 3 
C 


Grac 

E 


ie 7 
C 


Grac 

E 


ie 8 
C 


Grac 

E 


ie 9 
C 


Pretest 

Posttest 

Raw Score 
Gain 

GE Gain 


68 

(2)* 


74 

(3)* 


32 

(1.5) 


35 

(1.6) 


33 

(2.1) 


42 

(2.3) 


40 

(4.5) 


46 

(5.0) 


32 

(4.8) 


38 

(5.6) 


38 

(5.6) 


45 

(6.4) 


57 

(0.8) 


62 

(1.2) 


51 

(1.9) 


55 

(2.0) 


49 

(2.5) 


55 

(2.7) 


45 

(4.9) 


50 

(5.3) 


39 

(5.7) 


45 

(6.4) 


45 

(6.4) 


51 

(7.1) 


- 


- 


19 


20 


16 


13 


5 


4 


7 


7 


7 


6 




- 


0.4 


0.4 


0.4 


.0.4 


0.4 


0.3 


0.9 


0.8 


0.8 


0.7 



Sample 1047 1092 1242 1192 1317 1165 1211 1108 1151 1105 1093 996 

Size . ' 



* Stanine values 




±21 



108 



This is shown in the plot by the open-headed arrow having its initial and 
final points corresponding to the grade equivalent gain. The associated 
solid-headed arrow shows the grade equivalent gain for the control group. 
For third grade, mathematics, these pre and post grade equivalents for 
the control group are 2.3 and 3,0, respectively. In this case the 
experimental group gained, on the average, 0.4 grade equivalents, whereas 
the control group gained 0.7 grade equivalents. The standard gain in 
grade equivalents is equal to 1,0, and will occur whenever an arrow has 
a slope equal to that of the normal grade equivalent gain line shown in 
the plot. Slopes less than one indicate a relatively reduced rate of 
achievement; the smaller the slope the greater is the reduction in 
achievement rate. 

An examination of the figure shows that the experimental groups 
started at lower pretest levels than did the control groups for grades 
3, 7, 8, and 9, in both reading and mathematics. In no case are the 
arrows parallel to the standard progression line. More importantly, in 
no case is the slope of an arrow associated with the experimental group 
markedly greater than that of the arrow associated with the control 
group. In other words, the rate of achievement of the experimental 
groups is not improved to any noteworthy extent over that of the control 
groups. 
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Experimental Versus Control Groups ; 

Second-Year Stability of Impact Results 

Selection of Grade/Site/Sub ject 
Combination for Second-Year Testing 

In order to assess the stability of the impact of the experimental 
program over the summer months, certain grade/site combinations were selec- 
ted within which students from the experimental (E) and control (G) groups 
were given the reading and/or mathematics subtests of the evaluation test 
during the fall of the 1971-1972 school year,* The particular form of the 
evaluation test given was the same form as that given to these students dur- 
ing the original pretest administration in t, fall of the 1970-1971 school 
year, with the exception of Grade 1 students. Grade 1 students received the 
same test (and form) as they received during the spring 1971 posttest 
administration. 

The primary strategy for selecting grade/site/subject combina- 
tions for second-year testing was to make a judgement, on the basis of 
preliminary data analyses,** where a significant positive or negative 
difference between the E and C groups occurred on the basis of first 
year test results. This judgement was made by examining the differences 
between E and C mean raw score gains, relative to their estimated variance. 
Second year testing, then, was planned for those grade/site/subject combi- 
nations where the preliminary analyses indicated either a positive or 
negative impact. Both experimental and control students were to be tested 
in the selected grades, sites, and subjects. Finally, the students to be 
tested * during the second year had to be full year students during the first 
year, with both pre and posttest scores for the first year. 



* Specifically, five sites (Selmer, Las Vegas, Athens, Portland, Fresno) 
tested during the week of November 29, 1971 and four sites (Dallas, 

. Anchorage, Grand Rapids, and Jacksonville) tested during the week of 
December 6, 1971, 

** At the time when decisions had to be made concerning where second 

year testing was to occur, formal statistical regression analyses had 
not yet been accomplished. 
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In accordance with the above procedures, the grade/site/sub ject 
combinations for which second year testing occurred are identified in 
Table 32. Whereas there is some balance between subject area, most of 
the testing occurred in the Elementary grades. This result is consistent 
with the higher density of significant Pre-Post Model differences in favor 
of the E or C group in the Elementary grades than in the Junior High grades 
in the fir t year analyses. 

TABLE 32 . SITE /GRADE /SUBJECT COlffilNATIONS WHERE SECOND-YEAR TESTING 
WAS ACCOMPLISHED 



Site 


Grade 1 


Grade 2 


Grade 3 


Grade 7 


Grade 8 


Grade 9 


R 


j M 


R 


M 


R 


M 


R 


M 


R 


M 


R 


M 


Selmer 


X 




X 


X 


X 


X 














Athens 


X 


X 






X 


X 


X 


X 








X 


Dallas 


X 


X 




X 


X 


X 


X 




X 


X 






Anchorage 


X 


X 


X 


X 


X 








X 








Las Vegas 






X 


X 


X 


X 














Fresno 




X 




X 




X 

> 














Grand Rapids 
















X 


X 


X 


X 




Portland 




X 


* 


X 




X 






- 








Jacksonville 


X 


X 


X 


X 


X 


X 















R = Reading 
M = Mathematics 

X - Occurrence of second year testing 
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Stability of Impact Results 

j 

Table 33 shows the estimated means for which the difference was 
statistically significant for the experimental and control groups using the 
p re -post regression model to analyze the fall 1971 second-year test scores 
against the fall 1970 pretest scores. The table shows, for example, that 
in Grade 2, mathematics, at Dallas, the mean pretest raw score is equal to 
23, This value represents the mean raw pretest score for those students of 
the experimental and control groups who cook both the fall 1970 pretest and 
the second year test. The grade equivalent value that corresponds to this 
mean is shown in parentheses to be equal to 1,2, The estimated second-year 
test wan for the experimental group at the mean pretest value of 23 is 
shown to be 44 with a grade equivalent of 2,0, The corresponding estimated 
mean value for the control group is shown to be 31. with a grade equivalent 
value of 1,4. The difference between these regression estimates of second- 
year test means is given in units of raw scores and grade equivalents in 
the column labeled E/C Difference, In the present example, the raw score 
difference is seen to be +13 and the grade equivalency is seen to be +.6, 

Complete lists of all regression analysis results associated with 
the second year fall test are given in Appendix 8, whether such results were 
statistically significant or not, Also, associated t values are presented 
there. In addition. Appendix P provides pre- and second-year test means 
(and associated grade equivalents) and standard deviations, for each of the 
experimental and control groups. 

Table 34 illustrates the stability e r statistical outcomes of the 
Pre-Post Model analysis of spring 1971 posttest performance to fall 1971 
second-year test performance. In Grade 1 Reading, all outcomes remained 
unchanged. In Grade 1 Mathematics, one nonsignificant group difference 
from the spring analysis became significant in favor of the C group in the 
fall analysis; one E outcome become a nonsignificant outcome; two C's became 
N's; and only two outcomes, an N at Anchorage and an E at Jacksonville, re- 
mained the same. 
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TATLE 33, ESTIMATED EXPERIMENTAL (E) AND CONTROL (C) 
SECOND-YEAR TEST MEANS FOR SITES SHOWING A 
SIGNIFICANT GROUP DIFFERENCE 



Grade 

Subject 


Site 


Mean 

Pretest 

Value* 


Estimated 
Year Test 
E 


Second 

Means 

C 


E/C 

Difference 


Grade 1 


Selmer 


(3) 


81(2.2) 


73(1.8) 


+8 (+ , 4) 


Reading 


Dallas 


(1) 


63(1,2) 


73(1.8) 


-10(- .6) 




Jacksonville 


(2) 


66(1,5) 


57(0,8) 


+9 (+.7) 


Grade 1 


Athens 


(2) 


47(1.73 


53(1.9) 


-6(-.2) 


Math 


Jacksonville 


(2) 


46(1,7) 


34(1,0) 


+12 (+® 3 ) 


Grade 2 
Reading 


Jacksonville 


28(1.4) 


57(2.1) 


51(2.0) 


+6 (+, 1) 


Grade 2 


Dallas 


23(1.2) 


44(2,0) 


31(1,4) 


+13 (+, 6) 


Math 


Las Vegas 


21(1.2) 


37(1.6) . 


45(2,1) 


"8(- ,5) 




Jacksonville 


22(1.2) 


46(2,2) 


40(1.8) 


+6 (+.4) 




Portland 


40(1.8) 


54(2.7) 


56(3,1) 


"2 (~ .4) 




Fresno 


32(1.5) 


49(2.3) 


53(2.6) 


-4(-.3) 


Grade 3 


Selmer 


45(2,4) 


72(3.4) 


66(3,1) 


+6 (+ . 3 ) 


Reading 


Dallas 


27(1,8) 


40(2,3) 


34(2.2) 


+6 (+, 1) 




Las Vegas 


29(1.9) 


47(2.4) 


56(2,7) 


-9(-,3) 


Grade 3 


Selmer 


58(2.4) 


90(3,9) 


79(3,3) 


+11(+. 6) 


Math 


Dallas 


36(1.7) 


57(2,4) 


49(2.3) 


+8(+, 1) 




A.thens 


51(2.3) 


78(3.2) 


72(3.0) 


+6 (+.2) 




Jacksonville 


37(1,7) 


63(2.6) 


55(2.4) 


+8 (+.2) 


Grade 7 
Math 


Athens 


41(4.6) 


54(5.3) 


47(4.9) 


+7 (+.4) 


Grade 8 
Reading 


Grand Rapids 


32(4.8) 


43(6.2) 


37(5,5) 


+6 (+.7) 



* For Grade 1 the mean pretest value is given in staninesj all 
other raw score means in the table have their associated grade 
equivalents in parentheses. 
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A brief sumina ry of the number and kind of changes in Table 34 is 
presented in Table 35, This table shows a two-way classification of outcomes 
obtained for all of the 46 site/grade/subject combinations where second-year 
testing took place. The right-hand column of the table shows that 20 of the 
46 combinations yielded a significant impact for the experimental groups as 
a result of the spring post test analysis; 14 of the 46 combinations showed 
no impact as a result of the spring posttest analysis; and 12 of the combina- 
tions showed an impact in favor of the control group. The classification of 
impacts using the fall second-year test is shown by the bottom row of the 
table. This row shows 14, 27, and 5 impacts in favor of the experimental 
group, neither group, and the control group, respectively. 

An examination of the main diagonal of the table shows that 
13+12+4 - 29 of the 46 combinations yielded the same impact classification 
for both analyses; 17 combinations changed classifications between the 
spring and fall testing. Consequently, one measure of the stability of 
Impact is given by the ratio 29/46, or 63 percent. This gives an estimate 
of the unconditional probability that a random selection from these 46 com- 
binations will be classified the same way by the spring and fall analyses. 



TABLE 35. CLASSIFICATION OF RESULTS TO SHOW 
STABILITY OF PROGRAM IMPACT 



Classification* Based on 
Fall 1971 Second -Year Test 

E N C Total 



Classification* Based on E 13 7 0 20 

Spring (1971) Posttest N 1 • 12 1 14 

C 0 8 4 12 



Total 14 27 5 46 



* E denotes impact in favor of the experimental group 

C denotes impact in favor of the control group 

N denotes impact in favor of neither group 
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A more specialized examine tion of the E and C classifications can 
be made as follows, A total of 13 of the 20 combinations classified as E 
impacts by the spring posttests were also classified as E impacts by the 
fall tests. The ratio of 13/20 gives 65 percent as an estimate of the 
conditional probability that a randomly selected E-impact combination based 
on the spring posttest will also be classified as an E-impact combination by 
the fall second-year test, A total of only 4 of the 12 combinations classi- 
fied as C impacts in the spring analyses are so classified in the fall. The 
ratio of 4/12 gives 33 percent as an estimate of the conditional probability 
that a randomly selected C impact from the spring analysis will also be class! 
fled as a C Impact in the fall. It appears, then, that the outcomes in favor 
of the E group at the end of the 1970-1971 school year are more stable than 
the outcomes in favor of the C group. It also appears that the E impacts are 
more stable at the elementary than the secondary level, since only 4 of the 15 
elementary E impacts changed from spring to fall, whereas 4 of the 5 secon- 
dary E impacts changed from spring to fall. 

The nonsignificant group differences were the most stable of all. 
Twelve of the 14 N-impacts from the spring analyses remained the same; one 
became an E impact and one a C impact. 

Because some of the students who took the first- year tests were not 
available to take the second-year tests, the results presented above are 
based on a sample of students that is "reduced" relative to the "original" 
sample of first-year students. In general, approximately 30 percent of the 
students in the original sample were lost. In order to determine whether 
the mean pretest values for the original and reduced samples were essen- 
tially equal, examinations of these means were made for both groups,, These 
examinations indicated that over 90 percent of the 46 combinations that 
were posttested in the fall showed pretest means for the reduced sample 
within one raw score unit of the pretest mean of the original sample. Thus. , 
it is concluded that students in the second-year analysis are equivalent to 
students in the first-year analysis, with respect to original entry level 
achievement, so that any change in impact from one analysis to the other 
cannot be explained on this basis. 
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Comparison Groups Analysis 



In order to compare the performance of the experimental groups 
to the performance of students from within the same schools, the pool of 
replacement students at any grade/site was treated as a "comparison" (R) 
group for purposes of analysis. Comparison groups were posttested at any 
grade/site if there were judged to be at least 35 such students left at 
the end of the year. Table 36 identifies those grade/sites where compari- 
son group posttesting occurred. Results of applying the Pre-Post Model to 
the analysis of experimental versus comparison group differences, and to 
control versus comparison group differences, are presented and discussed 
below. Also, experimental versus control group impacts are related to 
comparison versus control group impacts. 



TABLE 36, SITES AND GRADES WHERE COMPARISON GROUPS WERE POSTTESTED 



Site 


Grade 1 


Grade 2 


Grade 3 


Grade 7 


Grade 8 


Grade 9 


Athens 






X 


X 


. X 


X 


Wichita 


X 


X 


X 


X 


X 


X 


Rockland 


X 


X 


X 


X 


X 


X 


Hartford 








X 


X 




McComb 


X 


X 


X 








Jacksonville 


X 


X 


X 


X 




X 


Hammond 


X 


X 


X 









X = Posttesting of comparison group 
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Control Versus Comparison Groups 

Table 37 shows a summary of the significant differences between 
the control groups and the comparison groups. Grades where comparison 
group testing was not done are indicated by a dash in the table. The symbols 
R and C signify that the regression estimates of the posttest means differ 
significantly in favor of the comparison (R) or control groups (C), respec- 
tively, No significant difference is indicated by the symbol N, A com- 
plete listing of the regression results is given in Appendix T, 

An examination of the table shows that comparison group testing 
was carried out for 58 grade/site/subject combinations. Among these com- 
binations a total of 19 significant differences were found between the 
regression estimates of the posttest means for the control and comparison 
groups. Thus, significant differences are shown for approximately one- third 
of the testing combinations. Among these 19 differences, 14 (74 percent) 
favor the comparison group. The table also shows highly consistent results 
for a given site. For example, at Athens, Georgia, six of these groups gave 
significantly higher estimated post test means for the comparison groups. In 
no case at Athens did the results favor a control group. The significant 
results at Rockland also favor the comparison groups, whereas the significant 
results consistently favor the control groups at Hammond and McComb, At 
Jacksonville, five results favor the comparison groups, and one result 
favors the control group. Although 12 comparison groups were tested at 
Wichita, no significant differences were found. 

Experimental Versus Comparison Groups 

Table 38 shows a summary of the significant differences between 
the experimental groups and the comparison groups. The symbols are the 
same as those shown in the preceding table except that the experimental 
group symbol £ replaces the symbol C associated with control groups. These 
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Grades where posttesting of comparison groups was not done are indicated by "a dash; the symbol R (or C) 
signifies that the regression, estimates of the posttest means favor the comparison group (or 
control) group; no significant difference is indicated by an N. 
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Grades where posttesting of comparison groups was not done are indicated by a dash; the symbol R (or E) 
signifies that the regression, estimates of the post test means favor the comparison (or experimental) 
group; no significant difference is denoted by an N. 



120 



differences were obtained from the complete regression results presented 
in Appendix U. In addition, Appendix 0 provides pre and post comparison 
group means (and associated grade equivalents) , along with a repeating of 
experimental group means, so that the interested reader can make comparisons 
between E and R groups in terms of pre and post means. 

Among the 58 grade/site/subject combinations shown in the table, 

13 favor the comparison groups, only one favors the experimental group, and 
the remaining 44 combinations favor neither group. The significant differ- 
ences at Athens, Rockland, and Wichita all favor the comparison groups. The 
only significant difference in favor of the experimental group occurs at 
McComb in Grade 1 Mathematics, 

Experimental Versus Control Group Impacts as 
Related to Comparison Versus Control . Group Impacts 

Table 39 shows ordered pairs of significant differences for the 
58 site/grade/subject combinations where comparison groups were posttested. 
Consider the symbol ER associated with Athens, Grade 3 Mathematics, The 
first letter of this symbol indicates that the E group is significantly 
favored over the control group in the E versus C regression* the second 
letter indicates that the R group is significantly favored over the control 
group in the R versus C regressions. In general, the first symbol can be E, 

N, or C corresponding, respectively, to the experimental group, neither group, 
or the control group being favored by the experimental versus control regres- 
sions, Similarly, the second symbol can be R, N, or G corresponding, respec- 
tively, to the comparison group, neither group, or the control group being 
favored by the comparison versus control regressions. The first symbol is 
obtained from Table 38; the second symbol is obtained from Table 37, 

An examination of this table shows that the experimental group 
is favored over the control group a total of 13 times (E occurs 13 
times as the first symbol). The comparison group is also favored 
over the control group a total of 13 times (R occurs 13 times as the 
second symbol). Thus, the comparison groups are just as likely to 
be favored over the control groups as the treatment groups. 
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neither group, or the control group being favored by the experimental versus control regressions; 
the second symbol of each pair is R, N, or C corresponding, respectively, to the comparison group, 
neither group, or the control group being favored by the comparison versus control regressions® 
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It is also interesting to note that in the 13 eases when the 
experimental group is favored over the control group, in 9 out of 13 
cases (69 percent), the comparison group is also favored over the 
control group. This possibly suggests the operation of a "school 
factor" in certain cases of an apparent experimental impact. 

Finally, control groups do better when paired against 
experimental groups than when paired against comparison groups. 

Thus, control groups are favored 14 times when paired against 
experimental groups, but control groups are favored only six times 
when paired against comparison groups. 
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Analysis of Special Treatment Groups 

At two of the eighteen sites, Grand Rapids and Hartford, a 
number of additional remedial education programs were in progress 
during the 1970-71 school year for students in schools other than 
those designated as experimental and control schools# At Grand 
Rapids, students from Grades 1, 2, 3, 7, 8, and 9 were participating 
in four other programs: At Hartford, students from Grades 1, 2, and 

3 were participating in two special treatment programs. Table 40 
identifies these programs by site and grade. These programs are de- 
scribed breifly below. Following this description, the results of ’ 

comparing the special treatment group with the control and experimental 
groups are presented and discussed. 

Description of Special Treatment Groups at Grand Rapids 

The Westinghouse Learning Centers (Grades 1-3) program was quite 
similar to WLC programs in the experiment. It departed from experimental 
treatments, however, in that the program was not in grades 7-9 and also 
the size of individual centers was smaller. In the WLC experimental 
programs, individual centers were designed to accommodate 100 children 
at a time* in the WLC "special treatment" groups, however, center 
capcity was 30-40 children. The composition of student enrollment was 
similar to that in the experimental schools. 

The Combined Motivations and Education Systems (COMES) 

(Grades 7-9) instructional program was heavily machine orientated. The 
staff was about 80 percent paraprofessional and the overall staff- 
student ratio was approximately 1/12, One class per week was held in 
"achievement motivation", in which motivational values and incentives 
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TABLE 40, SPECIAL PROGRAMS AT GRAND RAPIDS AND HARTFORD 

GRAND RAPIDS 



Program 


Grade 1 


Grade 2 


Grade 3 


Grade 7 


Grade 8 


Grade 9 


Reading 

Centers 




X 


X 


X 






COMES 








X 


X 


X 


Westinghouse 


X 


X 


. X 








Project Read 


X 


X 


X 









X - Grades where programs implemented. 



HARTFORD 



Program 


Grade 1 


Grade 2 


Grade 3 


Grade 7 


Grade 8 


Grade 9 


Waver ly School 


X 


X 


X 








Project 

CONCERN 


X 


X 


X 









X = Grades where programs were implemented. 
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were discussed. There were no other incentive or point system. Students 
were similar to experimental students in terms of underachievement. 

Racially 5 most COMES students were black, compared to a mixed population 
in the experimental program and predominantly white population in the 
control program. 

Project Read (Grades 1-3 - Reading Only) is a program developed 
by Behavior Research Laboratories (BRL) . .It used Sullivan curricular 
materials. Regular school staffing was employed. This included one 
paraprofessional in each classroom, in addition to the teachers, resulting 
in a staff ratio of approximately 1/13. (Paraprof essionals were included 
in this manner in all "inner city" classrooms, consequently control 
classes were staffed in the same way.) No hardware or external Incentive 
systems were used. Although students were in a special treatment group 
in reading only, they were tested in both subjects. 

Reading Centers (Grades 2, 3, 7 - Reading Only) was not a program 
per se; rather it consisted of the use of special remedial reading teachers 
to teach underachieving students in small groups in their schools. This 
instruction, unlike the case of all other programs in the experiment, was 
supplementary to the school's normal reading programs. Each teacher was 
responsible for 2 or 3 schools. Scheduling of classes varied both as to 
times per week and size of the group. On the average, sessions were held 
with individual children 2-3 times per week. 

Description of Special Treatment Groups at Hartford 

The Waverly School (Grades 1-3) program consisted of a newly 
opened elementary school using the "open space" concept. It drew 
students from the same area as the experimental and control schools. 
Enrollment was governed by normal policies of the Hartford School 
District and was not related to the experiment. Student characteristics 
were similar to those in the experimental and control schools, , In 
addition to the open space environment, individualized instruction and 
team teaching were used. No particular emphasis was placed upon hard- 
ware, other than might be expected in a new school and no external 
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incentive systems were in use, Paraprofessionals supplemented certifi- 
cated teaching staff. The overall a taff-student ratio was approximately 
1 / 11 , 

Project CONCERN (Grades 1-3) is a program to bus disadvantaged 
children from the same general area served by experimental and control 
schools to middle class, predominantly white schools in other sections of 
the city. These children received the normal instruction provided in 
most of the city’s schools. The overall staff -student ratio was approxi- 
mately 1/16, The program was started on an experimental basis and was 
considered to be operational during 1970-71, 

Control Versus Special Treatment Groups 
at Grand Rapids 

Table 41 shows the regression estimates of the post test means 
for which the difference was significant for the control and special 
treatment groups at Grand Rapids, Michigan, The table shows, for example 
that in Grade 1 the special treatment (ST) program, Project Read , yields 
69 for the regression estimate of the posttest mean corresponding to the 
pretest stanine value of 3, The regression line for the control group 
yields 60 for its estimated posttest mean. The difference between these 
two estimates is -9. in favor of the special treatment group, and is 
shown in the column labeled C/ST Difference, 

An examination of the table shows that four out of five of 
the significant differencas are in favor of the special treatment 
programs* Project Read, COMES, and the Reading Centers. Only in 
Grade 8 -Mathematics does the control group perform significantly better 
than the special treatment, COMES , These five significant differences 
were found in a set of 24 comparisons between special treatment groups 
and control groups. Complete results for all comparisons are given 
in Append ix V , 

Control Versus Special Treatment 
Groups At Hartford 

Table 42 shows similar results for comparisons between special 
taatment groups and control groups at Hartford, Connecticut, The 
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special treatment program at Waverly school shows significantly better 
pgjf formanc e than the control groups for Grade 1— Reading and for Mathe— 
matics in Grades 2 and 3, The reverse result is shown for Project 
CONCERN where the control group is estimated to yield a higher achieve- 
ment level for Grade 1 -Mathematics , The four significant differences 
shown in this table were found in a set of 12 such comparisons at 
Hartford, Connecticut, Complete results are given in Appendix W. 

Experimental Versus Special Treatment 
Groups At Grand Rapids 

Table 43 shows significant results obtained from the 
regression lines fitted to the data for the experimental groups and 
the special treatment groups at Grand Rapids, Michigan, The regression 
lines are evaluated at the mean pretest value to obtain the estimated 
posttest means. 

An examination of the table shows that in every case in which 
a significant difference occurs, the difference is in favor of the 
associated special- treatment programs i Project Read, Reading Centers , 
Westinghouse , or COMES, The ten significant differences are equally 
divided between reading and mathematics, and occur for Grades 1, 2, 3, 
and 7, Complete results on a set of 24 such comparisons at Grand 
Rapids are given in Appendix V. 

Experimental Versus Special Treatment 
Groups At Hartford 

Table 44 shows results similar to the preceding table for special 
treatment groups, Waverly School and Project CONCERN, at Hartford, Con- 
necticut, The table shows that the regression estimates favor the special 
treatment groups over the experimental groups in every case in which a 
significant difference is judged to occur. Higher achievement is 
shown for three cases in reading, three cases in mathematics. Complete 
results for an additional set of six comparisons are given in Appendix W, ” 




In addition to the regression results given in Appendices V and W for 
experimental vs, special treatment groups. Appendix Q provides a com- 
plete listing of pro and post means (and associated grade equivalents) 
and standard deviations. 
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TABLE 43. ESTIMATED POSTTEST MEANS' FOR EXPERIMENTAL (E) AND SPECIAL 

TREATMENT (ST) GROUPS FOR GROUPS SHOWING SIGNIFICANT DIFFERENCES 
AT GRAND RAPIDS , MICHIGAN 
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Summary of Results for the Special 
Treatment Groups , 

Table 45 provides a summary of the regression analysis 
results for the special treatment groups at Grand Rapids and Hart- 
ford, For any grade and subject area (reading or mathematics), 
the first column (labeled "E/ST") shows the results of comparing 
the experimental group with the various special treatment groups. 

Three symbols can appear; "E" for a statistically significant 
impact in favor of the experimental group, "ST" for a statistically 
significant impact in favor of the special treatment group, and 
"N" for no statistically significant difference between experi- 
mental and special treatment groups. 

The second column in the table (labeled "C/ST") , for 
a given grade and subject area, shows the results of comparing the 
special treatment and control groups. Again, three symbols can 
appear, "C" indicating a statistically significant impact in favor 
of the control group, n ST" for a significant impact in favor of the 
special treatment group, and "N" denoting no significant differences 
between special treatment and control groups, 

A dash in a given cell of the table indicates that the 
special treatment program was not implemented at that grade/sub ject 
combination. 

For two of the programs, Wes ting house and Project CONCERN, 

the general outcome is no significant difference, with only one or 

two significant impacts when compared against either the experimental 

or control groups. The COMES program exhibits this same pattern, but 

to a lesser degree. Comparing COPES to the control groups in Grades 7, 

8, and 9, 4 of the 6 comparisons yield no significant difference j and as 

compared to the experimental groups, again 4 of the 6 comparisons yield 
* 

no significant difference. 

On the other hand, the Waverly school program at Hartford 
shows promise of enhancing students 1 achievement. Of the 12 comparisons 
of this program with the experimental and control groups, in 8 of 
these 12 there was a statistically significant impact in favor of 





TABLE 45. SUMMARY OF RESULTS FOR THE 
SPECIAL TREATMENT CROUPS* 
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* E denotes an impact in favor of the E group; C an impact in f av „r of tine C group; ST an impact in favor of the 
special treatment group; N an impact in favor of none of the groups. 

** Reading only* taught in the special treatment program. 
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the Waverly school program. In 5 of the 6 comparisons with the 
experimental group, Waverly was favored, with 1 of the comparisons 
yielding no significant difference; in the 6 comparisons with the 
control group, Waverly was favored in 3, with 3 yielding no signifi-- 
cant difference. In all four cases of no significant difference, 
however, the estimated differences in posttest means were nonetheless 
in favor of the Waverly program (see Appendix W) , but not reaching 
statistical significance. 

Thus, there is evidence that the Waverly program enhances 
student achievement, when compared against either the experimental 
or control groups, but more evidence when compared against the experi 
mental, group. It is interesting to note that the Waverly program 
does not emphasize hardware, and that no external incentive systems 
were in use, as discussed earlier. Rather, the focus was on the 
"open space" concept, emphasizing individualized instruction and 
team teaching. 

The Reading Center program, and Project Read, both at 
Grand Rapids, also exhibit some evidence of having a positive 
effect on student achievement. Although, in both of these programs, 
only 5 of the 12 comparisons for each program were statistically 
significant in favor of the special treatment, for the remaining 
7 comparisons, the regression estimates generally favored the special 
treatment programs (see Appendix V), That is, in these cases, the 
estimated difference in post test means generally favored the special 
treatment group, but this difference was not large enough to reach 
the specified level of significance. It is again interesting to 
note that in both of these programs, there was no emphasis on hard- 
ware, and no external incentives were used. It is also interesting 
to note that in both of these programs, remedial instruction was 
given in reading only, as mentioned earlier. 

In terms of comparing overall the experimental programs 
with the special treatment groups, the most conspicuous result shown 
in Table .45 is the absence of any positive impact in favor of the 
experimental treatment, out of 24 such comparisons in Grand Rapids, 
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and out of 12 such comparisons in Hartford, However, the special 
treatment group was favored in 10 of the 24 comparisons at Grand 
Rapids, and in 6 of the 12 comparisons at Hartford, 

In terms comparing overall the control groups with the 
special treatment groups, the special treatment groups did not fare 
as well as when compared with the experimental groups. Out of 24 
special treatment-control group comparisons in Grand Rapids, 4 yielded 
a significant impact in favor of the special treatments, with only 
1 impact in favor of the controls. Out of the 12 special treatment- 
control group comparisons at Hartford, 3 yielded a significant 
impact in favor of the special treatments, and again with only 1 
significant impact in favor of the controls. 
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Analysis of Attendance Data 



In lieu of any variables which measure student motivation and 
attitudes toward school and learning, data were gathered on regular school 
attendance for students in the experimental and control groups wherever 
possible for the school years, 1970-71 and 1969-70 (the latter being 
considered as a "pre" measure or "entry level"). Attendance can be 
construed as at least an indicator of a student's willingness and 
desire to learn. The grade/sites for which data were collected for 
a Pre-Post Model analysis of attendance are identified in Table 46, The 
results of these analyses are presented and discussed here. Note that 
the attendance variable is expressed in units of percent days absent . 



TABLE 46. SITES AND GRADES WHERE ATTENDANCE ANALYSES WERE PERFORMED 



Site 


Grade 1 


Grade 2 


Grade 3 


Grade 7 


Grade 8 


Grade 9 


Selmer 




X 


X 


X 


X 


X 


Athens 




X 


X 


X 


X 


X 


Wichita 




X 


X 




X 


X 


Dallas 




X 


X 


X 


X 


X 


Rockland 


X 


X 


X 


X 


X 


X 


Las Vegas 








X 


X 


X 


Fresno 




X 


X 


X 


X 


X 


McComb 




X 


X 


X , 


X 


X 


Seattle 




x 


X 


x- 


X 


X 



X = Attendance analysis performed 
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Table 47 shows the estimated 1970-71 percentage of school 
days absent for the experimental and control groups that showed 
significant differences between their rates of absenteeism. The 
attendance data were analyzed by regressing the percentage of days 
absent for each student during the 1970-71 school year against his 
percentage of days absent in 1969-70, as calculated from attendance 
records. The table shows, for example, that in the second grade at 
Dallas, the median* 1969*70 percentage of days absent for the combined 
experimental and control groups is equal to 10,25 percent. During the 
program, the estimated percentage obtained from the regression line 
fitted to the data for the E group is shown to be 1,29 percent. This 
value corresponds to the median value of 10,25 percent. The regression 
line, evaluated at 10,25 percent, yields an estimated mean of 7.72 
percent for the C group. Thus, the absenteeism of the E group is 
estimated to be approximately 6,43 percentage points less than that of 
the C group, as shown in the column labeled E/C Difference. As 
show in Table 47, a reduction in absenteeism of the E group over the 
C group occurred six times (Dallas, Grades 2, 7, and 8; Athens, Grades 
8 and 9; and Selmer, Grade 9), These 6 cases may be compared with 
the treatment results shown in Table 27. Table 27 shows that the 
six grade/site combinations found to favor the E group on attendance 
yield four favorable and one unfavorable impact of the experimental 
treatment on posttest performance. Table 27 also shows that no favor- 
able impacts and three unfavorable impacts on posttest performance 
were obtained for the four grade/site combinations shown in Table 47 
that were found to favor the C group on attendance. 

A complete listing of the attendance results is found in 
Appendix X, Among the 48 grade/site combinations found there, 24 
show positive E/C differences and 24 show negative E/C differences. 



* The median was used because the distribution of absenteeism was highly 
skewed. 
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TABLE 47. ESTIMATED MEAN 1970-71 PERCENTAGE OF SCHOOL 

DAYS ABSENT FOR EXPERIMENTAL (E) AND CONTROL (C) 
GROUPS SHOWING SIGNIFICANT DIFFERENCES 



Median 69-70 Est, Mean 70-71 
Percentage of Percentage E/C 

Grade Site Days Absent E C Difference 



2 Dallas 

Rockland 

7 Dallas 
Fresno 

8 Dallas 
Athens 
McComb 

9 Selmer 
Athens 
McComb 



10.25 


1.29 


5.65 


6.89 


8.70 


7.14 


5,16 


10,10 


5.97 


5,83 


6.17 


5.50 


3,97 


7.35 


4.00 


2.03 


7,36 


5,84 


3.02 


8.33 



7.72 


-6 , 43 


5,10 


1.79 


10.62 


-3.48 


6,93 


3,17 


14.12 


-8.29 


8.93 


-3.43 


4,76 


2,68 


4,96 


-2,93 


10.68 


-4.84 


3.10 


5,23 
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Thus, in general, it appears that the experimental treatment did not 
reduce absenteeism. However, wherever absenteeism was relatively low 
for the experimental groups vs. the control groups, the data suggest 
the possibility of enhanced experimental treatment impact on postfest 
performance. 
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Summary of Results and Conclusions 

In this section a brief summary is given of the principal results 
presented in previous sections. Conclusions based upon these results are 
presented at the end of the section. 

Sunmarv of Principal Results 

9 Based upon 212 pre-post regression analyses of 
first year posttest performance in reading and 
mathematics of experimental and control 
students, 28 group differences (or 13 percent) 
were found in favor of the experimental group, 60 
group differences (or 28 percent) were found in 
favor of the control group, and 124 group differences 
(or 59 percent) were not significant. In no grade, 
subject, or grade/subject combinations did the number 
of impacts favoring the experimental group exceed the 
number of impacts favoring the control group, 

. • Based upon 184 regression analyses involving 

' extended regression models* again comparing the 
posttest performance in reading and mathematics 
of experimental and control students, 15 group 
differences (or 8 percent) were found in favor of 
the experimental group, 27 group differences (or 15 
percent) were found in favor of the control group, 
and 142 group differences (or 77 percent) were not 
significant. Thus, the extended regression models 
yielded even a smaller percentage of impacts in 
favor of the experimental group than did the pre- 
post regression analysis. 
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• An overall comparison of the pre-post achievement 
gains of the experimental and control groups 
showed almost identical gains for the two groups, 
in reading and in mathematics, and at each grade 
level . 

• Based upon 58 pre-post regression analyses of the 
experimental versus comparison group posttest 
performance in reading and mathematics, 

13 significant differences were found in favor of 
the comparison group, and only one in favor of the 
experimental group. 

• Based upon 46 pre-post regression analyses of 
second year test performance in reading and mathe- 
matics of experimental and control students, 13 
(or 65 percent) of 20 first-year impacts in favor 
of the experimental program were maintained as 
measured by second- year test results. 

• Of the 43 applications of the Pre-Post Model to 
attendance, comparing the experimental and control 
groups, 10 differences were significant. Six 
differences showed a lower rate of absenteeism 
for the experimental group than for the control 
group. 

• Of the 24 pre-post regression analyses of experi- 
mental versus special treatment groups in reading and 
mathematics at Grand Rapids, 10 group differences 
were found to be significant. Of the 12 analyses of 
special treatment groups at Hartford, 6 group 
differences were found to be significant. All 16 



significant differences were in favor of the special - 
treatment groups. 
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Conclusions 



Several analyses of the data were performed, directed toward assess- 



ing the Impact of performance incentive contracting on student achievement. 
Analyses were conducted at each slte/grade/subject combination, as well as 
in the aggregate across several sites, and involved comparisons of several 
different groups with the experimental group. 



As explicitly suranarized above, all analyses point toward one out- 



come: the small amount of evidence in favor of the experimental groups. 

Tills outcome, however, must be evaluated in the light of three important 
points. First, the analysis evaluated performance incentive contracting 
as implemented by the six technology companies involved in the study . The 
evaluation makes no pretense of generalizing results and conslusions to the 
effectiveness of performance incentive contracting in general. That is, 
although efforts were made to select technology companies with widely varying 
educational approaches to implement their programs in a variety of locations, 
results and conclusions from this study are not to be generalized to the 
effectiveness of performance incentive contracting as implemented by other 
technology companies in other school districts. Indeed, the obtained 
outcomes for the six companies involved in the study could he different if 
they were to implement their program in a different way as a result of 
experiences gained during the experimental year. 



as implemented for a period of one year . The evaluation makes no pretense 
of generalizing results and conclusions to the effectiveness of performance 
incentive contracting implemented for more than one year. 



using a standardized achievement test as the basis for assessing program 
impact. The evaluation makes no pretense of generalizing results and 
conclusions to the effectiveness of performance incentive contracting using 
some other method of assessing program impact. 



Accordingly, the following conclusion is drawn: 

• There is very little evidence that performance incentive 
contracting, as implemented by the technology companies 
at the 18 school districts in this study for a period of 
one year, had a beneficial effect on the reading and 
mathematics achievement of students participating in the 
experiment, as measure’ ' a standardized achievement test. 



Second, the analysis evaluated performance incentive contracting 



Finally, the analysis evaluated performance incentive contracting 
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RESULTS AND CONCLUSIONS 
INCENTIVES ONLY SITES 



In Batte ' .-Columbus 1 Interim Report on the Incentives Only 
sites (February 7, 1972)*, it was stated there that this interim 
report would constitute the final report on the Incentives Only 
sites, with the exception of reporting the results of additional 
analyses on data from these sites in this final report. It is the 
purpose of this section to review the results previously reported, 
and to present the results of additional analyses employing the 
Extended Variables Model, the Pre-Post Model with second year test 
scores in order to study stability of impact, and the Pre-Post Model 
applied to attendance data. For information on descriptions of the 
experimental programs at Stockton and Mesa, a description of the 
target population, reports on testing conditions, and all statistical 
appendices, the reader Is referred to Battelle-Columbus 1 previous 
Interim Report, 



Experimental Versus Control Groups* 

Regression Analyses of First-Year Results 

The purpose of this section is to present the principal 
results based upon tests of significance of the differences between 
the experimental (E) and control (C) groups’ estimated posttest means. 
As mentioned earlier, the analysis is confined to full-year students. 
Also, the analysis includes only those full-year students who have 

both a pretest and posttest score, 

* 

Pre-Post Model Results 



Results are presented for reading and mathematics within 
each grade. Tables, and descriptive discussions of them, show the 



* Op, cit. , page 13 
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grade/site/subject combinations for which statistically significant 
differences were found, the estimated posttest means for each group, 
their differences, and the combined E and C pretest mean associated 
with the posttest differences. All analyses are done with raw scores J 
however, the grade equivalents of the raw scores are presented also 
to facilitate the interpretation of the results through the use of 
a commonly used metric. 

Table 48’ shows the results obtained for Mesa and Stockton 
for those grade/subject combinations having a statistically significant 
group difference. As indicated by the table, for Grade 1, significant 
group differences occurred only in reading at Stockton, The difference 
found favored the experimental group. The table shows that a difference 
of +10 in the posttest means for Grade 1 reading is obtained when the 
fitted regression equations are evaluated at the combined mean of 
the pretest raw scores for the E and C groups. This combined mean is 
shown in Column 2 to have a stanine value of 4 and the individual 
regression estimates for the E and G groups are shown as 80 and 70 
in Columns 3 and 4, respectively. The grade equivalents corresponding 
to 80 and 70 are shown in parentheses as (2.2) and (1,7), respectively, 
and the cifference between these grade equivalents is shown as (+.5) 
in Column 5, 

For Grade 2-Mathematics, the table shows that both Mesa and 
Stockton yielded significant differences between the posttest means 
when the regression lines were evaluated at the mean pretest value of 
31, These posttest differences are seen to be +3 and +4 for Stockton 
and Mesa, respectively, and in both cases the differences favored the 
experimental group. 

For Grade 2-Reading, no significant differences were found 
in either Mesa or Stockton. Likewise, for Grades 3, 7, and 8, no 
significant differences were found in Mesa or Stockton, in either 
reading or mathematics. 

For Grade 9-Reading, Table 48 shows a significant negative 
impact for both Mesa and Stockton with a ( difference of -5 for the 
posttest means in each site as estimated by the regression equations. 

The grade equivalents show a gain from pretest to posttest of 1,2 for 
the control group at Stockton, 



158 



145 



TABLE 48, ESTIMATED EXPERIMENTAL (E) AND CONTROL (C) POSTTEST MEANS AND 
DIFFERENCES BETWEEN TlffiSE MEANS FOR THOSE GRADE /SITE /SUBJECT 
COMBINATIONS EXHIBITING A SIGNIFICANT GROUP DIFFERENCE 



I 



READING 



GRADE 1 





Mean 

Pretest 


Estimated 
Posttest Means 


E/C 


Site 


Value* j 


E 


C 


Difference 


Stockton 


( 4 > i 

j 


80 (2.2) 


70 (1.7) 


+10 (+.3) 



* The stanine value corresponding to the combined E and C 
pretest raw score mean. 



GRADE 2 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E 


C 


Stockton 


31 (1,4) 


49 (2.3) 


46 (2.2) 


+3 (+.1) 


Mesa 


31 (1.4) 


43 (2.0) 


39 (1.7) 


+4 (+.3) 



GRADE 9 



READING 



Site 


Mean 

Pretest 

Value 


Estimated 
Posttest Means 


E/C 

Difference 


E 


C 


Stockton 


35 (5.2) 


39 (5.7) 


44 (6.4) 


-5 (-.7) 


Mesa 

— — 


46 (6.6) 


48 (6.7) 


53 (7.3) 


-5 (-.6) 



i. 

i 
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Summary of Pre-Post Model Results 

Table 49 shows a concise suiranary of the principal results 
based upon the Pre-Post Model for Mesa and Stockton* The symbols , E and 
C, indicate the occurrence of a positive or negative difference, 
respectively, between the posttest means as estimated by the regression 
lines fitted to the pre-post data among those sites having a significant 
group difference. An empty cell indicates that no significant difference 
was found between the estimated posttest means for the E and C groups. 

An examination of the table shows that Mesa had one positive 
impact (Grade 2 -Mathematics) and one negative impact (Grade 9-Reading), 
Stockton had two positive impacts (Grade 1-Reading and Grade 2 -Mathematics) 
and one negative impact (Grade 9-Reading), For Stockton and Mesa 
combined, the total number of positive impacts is 3; the number of 
negative impacts is 2; and the number of instances showing no signifi- 
cant differences is 19, 



TABLE 49, SUMMARY OF PRE-POST MODEL RESULTS FOR EACH 
GRADE, AND SUBJECT AT MESA AND STOCKTON 



ELEMENTARY 





4 

Grade 1 


Grade 


j 

2 


Grade 3 


Site 


Read ing 


Math 


Reading 


Math 


Reading 


Math 


Mesa 








E 






Stockton 


E 






E 








. SECONDARY 





Gradi 


a 7 


Grade 


8 


Grade 9 


Site 


Reading 


Math 


Reading 


Math • 


Reading 


Math 


Mesa 








• 


C 




Stockton 










C 





iso 



Extended Variables Model Results 



The Extended Variables Model was also applied in the analyses 
of the Incentives Only sites,. Table 50 indicates the variables in 
this model for both sites. 



TABLE 50. INDEPENDENT VARIABLES CONSTITUTING THE EXTENDED 

VARIABLES MODEL FOR EACH GRADE/SITE COMBINATION * 



Site 


Grade 


Group 


Independent Variables 

Student's Father's 
Pretest Race Education 


Total Family 
Income 


Approve 
New Method 


Stockton 


All 


X 


X 


X 


X 


X 


X 


Mesa 


2 


X 


X 


X 


X 




X 


Mesa 


8 


x 


X 


X 








Mesa 


1.3, 7, &9 


X 


X 


X 


X 


X 


X 



* Presence of variable in IVM is indicated by an "X", 



Table 51 shows the results of changing from the Pre-Post 
Model. to an Extended Variables Model for Stockton and Mesa. In the 
elementary grades , only the difference in favor of the E group in 
Grade 1-Reading at Stockton was maintained. In the junior high grades, 
the two differences in favor of the C group in Grade 9-Reading, 
remained. Also, a difference in favor of the C group was obtained 
at Mesa with the EVM in Grade 7-Mathematics, where it had not been 
found with the PPM, 
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TABLE 51. COMPARISON OF SIGNIFICANT E/C GROUP DIFFERENCES FOUND 
USING THE PRE-POST MODEL VERSUS Tlffi EXTENDED VARIABLES 
MODEL FOR EACH GRADE, SITE, AND SUBJECT 



ELEMENTARY 



Site 


Grade 

Reading 


! 1 

Ma 


th 


Grade 2 

Reading Ma 


th 


Reac 


Grade 3 

ing Math 


PPM 


EVM 


PPM 


EVM 


PPM 


EVM 


PPM 


EVM 


PPM 


EVM 


PPM 


EVM 1 


Stockton 


E 


E 










E 












Mesa 














E 













JUNIOR HIGH 



Site 


Rea< 


Grade 7 

ling Math 


Reac 


Grade 8 

Ling Math 


Reac 


Grade 9 
ing Math 


PPM 


EVM 


PPM 


EVM 


PPM 


EVM 


PPM 


EVM 


PPM 


EVM 


PPM 


EVM 


Stockton 


















C 


C 






Mesa 








G 










C 


C 







Stability of Impact Results 

Second-year testing was carried out in Grades 2 and 3 at 
Mesa only. * There was one significant group difference found in these 
two grades based upon first year analyses and it was in favor of the 
experimental group in Grade 2-Mathematics. As Table 52 shows* this 
impact did not remain. No other impacts were found. 

Analysis of Attendance Data 




Application of the Pre-Post Model to attendance data was 
possible at Mesa in Grades 2, 3* . 7, 8, and 9, As Table 53 shows* 
no significant differences between the experimental and control 
groups were found, 

* Testing was accomplished during the week of November 29, 1971. 

1.62 
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TABLE 52 . REGRESSION RESULTS FOR SECOND YEAR -TESTING AT MESA 

GRADE 2 



Site 


Mean 

Pretest 

Value 


Estimi 

Posttest 


ited 
: Means 


E/C 

Difference 


t 

Ratio 


E 


C 


Reading 

Mathematics 


31 (1.5) 


56 (2.0) 


53 (2.0) 


+3 (0) 


1.019 


32 (1.5) 


44 (2.0) 


44 (2.0) 


0 (0) 


-0.273 



GRADE 3 



Site 


Mean 

Pretest 

Value 


Estimi 

Posttes( 


ited 

Means 


E/C 

Difference 


t 

Ratio 


E 


C 


Reading 

Mathematics 


39 (2.3) 


57 (2.8) 


60 (2.9) 


-3 (-.1) 


-0,985 


48 (2.3) 


64 (2.7) 


66 (2,8) 


-2 (-.1) 


-0.737 



TABLE 53 REGRESSION RESULTS FOR ATTENDANCE AT m,SA 





Med ian 


Estimated Mean 








69-70 


70-71 At 


tendance* 


E/C 


t 


Grade 


Attendance* 


E 


C 


Difference* 


Ratio 


2 


6.98 


6.73 


5,73 


+1,00 


0,993 


3 


7.74 


7,39 


7.15 


+0,25 


0,301 


7 


6.12 


7.56 


6.93 


-0,63 


0,582 


8 


6.27 


5.95 


7.86 


-1.91 


-1.286 


9 


7,32 


7.73 . 


9,53 


-1,80 


-1.494 



* All entries are percentage days absent 



ERIC 



163 



150 



Summary of Results and Conclusions 

Summary of Results 



• Based upon analyses in Grades 3* 7, 8, and 9, there 

is no evidence that the incentive programs at Stockton 
and Mesa had a beneficial effect on students' reading 
and mathematics standardized test achievement, 

» Based upon analyses in Grade 2, there is little evidence 
that the incentive programs at Stockton and Mesa had 
a beneficial effect on students’ mathematics standardized 
test achievement. The apparent impacts in Grade 2 mathematics 
at Stockton and Mesa in favor of the incentive programs were 
not found to be persistent under varying methods of analysis. 

For Grade 2 reading* there is no evidence that the incentive 
programs had a beneficial effect on students' reading 
standardized test achievement, 

• Based upon analyses in Grade 1, there is some evidence 
that the incentive program at Stockton had a beneficial 
effect on students' reading standardized test achievement, 

but not at Mesa, For Grade 1, mathematics, there is no evidence 
at either Stockton or Mesa that the incentive programs had 
a beneficial effect on students' standardized test 
achievement . 



Conclusion 



drawn: 



Based on the above results the following conclusion is 

• Overall, there is little or no evidence at Stockton 

and Mesa that the "Incentives Only" programs were beneficial 
to the students in reading or mathematics achievement, 
as measured by a standardized test, . 
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APPENDIX A 



TABLE A- 1. NUMBER OF FULL-YEAR STUDEr 3 
BY SITE, GROUP, AND GRADE 




1 2 3 7 8 ,9 



Site 


Exp 


Con 


Exp 


Con 


Exp 


Con 


Exp 


Con 


Exp 


Con 


Exp 


Con 


Anchorage 


50 


62 


53 


54 


56 


57 


51 


71 


51 


69 


58 


53 


Athens 


78 


82 


78 


94 


87 


80 


85 


79 


81 


75 


77 


76 


Bronx 


84 


101 


67 


107 


86 


94 


68 


123 


82 


82 


65 


83 


Dallas 


87 


86 


89 


89 


88 


89 


83 


75 


89 


84 


86 


88 


Fresno 


81 


89 


87 


92 


91 


91 


87 


87 


91 


86 


80 


83 


Gr, Rapids 


82 


;a 


91 


80 


92 


77 


92 


85 


87 


89 


84 


71 


Hammond 


84 


111 


77 


108 


85 


134 


96 


101 


92 


100 


88 


98 


Hartford 


70 


93 


69 


93 


74 


76 


79 


70 


95 


90 


93 


126 


Jackson 


93 


94 


95 


89 


93 


86 


88 


86 


94 


91 


85 


90 


Las Vegas 


69 


47 


66 


51 


82 


34 


83 


54 


84 


52 


71 


55 


Me Comb 


72 


47 


71 


44 


71 


53 


72 


55 


75 


47 


73 


66 


Phi la. 


117 


58 


111 


113 


102 


84 


52 


67 


65 


60 


58 


46 


Portland 


84 


100 


89 


102 


89 


100 


88 


97 


84 


93 


73 


94 


Rockland 


101 


80 


91 


79 


112 


82 


102 


73 


100 


68 . 


87 


75 


Seattle 


75 


80 


79 


39 


82 


92 


88 


94 


84 


96 


89 


95 


Selmer 


94 


38 


100 


46 


103 


47 


94 


54 


88 


52 


98 


40 


Taft 


64 


83 


67 


99 


75 


95 


65 


79 


51 


85 


47 


51 


Wichita 


86 


60 


83 


73 


81 


60 


82 


95 


85 


84 


89 


93 




APPENDIX B 



TABLE B-l SURVEY RETURN RATE BY SITE, 
GROUP, AND GRADE 



Site 










■ ■ 


Grade 








1 1 






1 




2 




3 




7 




8 




9 


Exp 


Con 


Exp 


Con 


Exp 


Con 


Exp 


Con 


Exp 


Con 


Exp 


Con 


Anchorage 


43 


52 


41 


58 


49 


61 


34 


56 


31 


65 


40 


51 


Athens 


68 


63 


72 


74 


68 


68 


48 


69 


60 


59 


58 


63 


Bronx 


26 


18 


20 


23 


23 


21 


13 


8 


23 


7 


0 


0 


Dallas 


68 


44 


71 


42 


69 


31 


50 


47 


38 


* 


41 


63 


1 iresno 


63 


66 


69 


66 


68 


62 


42 


40 


26 


36 


29 


40 


Gr. Rapids 


59 


45 


46 


42 


52 


50 


52 


49 


35 


50 


42 


34 


Hammond 


52 


47 


53 


53 


50 


46 


49 


46 


38 


46 


36 


64 


Hartford 


14 


31 


26 


37 


16 


42 


26 


30 


22 


37 


37 


30 


Jackson. 


82 


68 


75 


75 


70 


69 


84 


67 


75 


70 


72 


70 


Las Vegas 


45 


9 


46 


34 


30 


35 


37 


54 


34 


30 


25 


29 


Me Comb 


62 


38 


51 


46 


52 


37 


44 


58 


64 


43 


51 


57 


Phil a. 


23 


0 


22 


0 


27 


0 


9 


17 


15 


33 


4 


19 


Portland 


51 


52 


52 


59 


47 


56 


46 


64 


44 


49 


29 


68 


Rockland 


76 


0 


64 


0 


59 


0 


46 


0 


47 


0 


34 


0 


Seattle 


48 


60 


45 


60 


50 


56 


40 


42 


39 


31 


35 


40 


Selmer 


60 


61 


64 


57 


67 


59 


60 


49 


65* 


68 


56 


49 


Taft 


44 


20 


28 


24 


37 


15 


25 


15 


23 


22 


23 


21 


Wichita 


56 


45 


64 


50 


70 


40 


72 


78 


54 


80 


58 


57 



* Dallas eighth grade produced a percentage return rate greater than one hundred. 
This can be only accounted for in terms of Dallas eighth grade adding more 
control students after the initial master list was created. The situation at 
Dallas was probably replicated at other sites; hence response rates should be 
interpreted cautiously. 



APPENDIX C 



PARENT QUESTIONNAIRE 




i 

! 

i 

i o 
ERIC 

.hffliflaffHMiaaa 



/ 



/ 



Battelle M( iinorial Institute • columbus laboratories 

505 KINO AVENUE COLUMBUS, OHIO 43201 • AREA CODE 614. TELEPHONE 299-3151 * CABLE ADDRESS: BATMIN 

December, 1970 



Dear Parent: 



NATIONAL SCHOOL SURVEY 

The purpose of these questions is to find out what parents think 
and feel about their children's schools and their children's progress in 
school. 



We need your help in learning about how parents feel about their 
schools and their children's schoolwork so that changes can be made to 
improve education. Would you please take a few minutes to answer the 
questions in this survey? We need every parent's help. About 30,000 
parents throughout the country are being asked these questions. After you 
have answered the questions, seal the form in the envelope and mall the 
questionnaire to Battelle Memorial Institute, Columbus Laboratories, in 
Columbus, Ohio, 

Only the survey group in Columbus, Ohio, will see your answers 
to the questions. No one in your school district--teachers , principals, 
or anyone else — will see your answers. Your answers to the questions are 
confidential ! 

Thank you for your help. 

Sincerely, 

Mt CJkkoJtT" 

Dr. Kenneth W« Eckhardt 
Educational Analysis 
Battelle Memorial Institute 
505 King Avenue 
Columbus, Ohio 43201 
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DIRECTIONS 

Please circle the number of the statement which is closest to your answer. 

Then write the number of your answer in the box at the right. 

EXAMPLE 

Should I circle the number of the answer I choose and write the number in the box? 

Yes ... © 

No . , , 2 



nj 



THE FOLLOWING QUESTIONS ASK ABOUT YOUR FEELINGS TOWARD EDUCATION IN GENERAL 



4. 



5. 



o ~ 

ERIC 



9 • I • ( 



1. How good a job are the public schools doing in educating children? 

A good job for all of the children 

A good job for about half of the children .... 

A good job for only a few of the children 

Doesn't do a good job for any of the children, , , , 






* • • 



i * • 



• * • 



• Pi 



I • I • I 



• it* 



How Important is It for a c 
Very important , , , . 
Somewhat important , 



to get as 

. . . .1 



schooling as possible? 
Not too important , 
Not important at all 



Do you agree or disagree: Public school principals care what parents think? 

Agree very much 1 Disagree somewhat , , 

Agree somewhat 2 Disagree very much , . 

Do you agree or disagree: Public school teachers c are what parents think? 

Agree very much 1 Disagree somewhat . , 

it somewhat 2 Disagree very much , , 



i ■ i ? i 



Do you feel parents have enough influence over what the schools do? 
Yes 1 No 



I « # I 



• ■ ■ 



ZZ7 

ZZ7 

ZZ7 

o 

ZZ7 



0, When children don't do well at school, which one of the following is most at fault? 

Classes are too large i 

Teachers don't know how to teach 2 

School work is too hard 3 

The children don't try 4 

Parents don't help their children enough . 



• •••••■■• 



7. Do you agree or disagree: School rules and regulations are too hard for parents 
to understand? 

Agree very much 1 Disagree somewhat 3 

Agree somewhat , 2 Disagree very much 4 



o 

o 



THE FOLLOWING QUESTIONS ASK ABOUT YOUR FEELINGS TOWARD YOUR LOCAL SCHOOLS 



8. Do you think that schools should be. trying new teaching methods? 
Yes * 1 No 



■ * * « 



2 



9. As far as you know, has your local school tried anything new in how children 
are taught? 

Y es 1 No ,i. , , I..*. 2 



o 

o 
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10 . 



11 . 



Would you approve or disapprove of your children being taught by a new teaching 
method? 



Approve very much 1 Disapprove somewhat . . 

Approve somewhat 2 Disapprove very much. . 



Is the child who brought this questionnaire home being taught by new 
teaching methods? 



Yes 



. . 1 



No. 






. 3 
. 4 



. 2 




12. From what you have seen or heard, do you think the new teaching method will 
be a success? 

My child is not being taught by any new method . 1 

Yes, the new method will be very successful for my child 2 

Yes, the new method might be successful for my child 3 

No, the new method won't be successful for my child 4 

13. Which kind of children do you think your child's teachers are most interested 
in teaching? 

Children who are fast learners 1 

Children who are average learners 2 

Children who are slow learners 3 

All of the children 4 

None of the children 5 





14. How are your local schools do ; ng in the following areas? Circle a number for 



=j rL! 

Excellent 


Good 


Fair 


Poor 


Job 


Job 


Job 


Job 


a) School rules and regulations. . , ^ 


2 


3 


4 


b) School subjects taught ^ 


2 


3 


4 


c) Teacher's ability to teach .... ^ 


2 


3 


4 


d) Methods of teaching courses . . ^ 


2 


3 


4 



How satisfied are you with the local schools? 

Very satisfied 1 

Satisfied 2 



Dissatisfied 

Very dissatisfied 






I I I I 



. 3 
. 4 






1 



ZZ7 

EJ 

EJ 

EJ 

EJ 



16. How often have you attended PTA or other meetings at your local school in the 
last year? 

Four or more times last year 

Two or.three times last year 

Once last year 

Never, last year *. 4 



EJ 



THE FOLLOWING QUESTIONS REFER TO THE CHILD WHO BROUGHT THIS QUESTIONNAIRE HOME FROM SCHOOL 



17. Compared to most children how well is your child doing in school? 

Better than most children in the same grade 1 

As good as most children in the same grade 2 

Not as good as most children in the same grade 3 



18. Does your child get help in doing school work from someone in the family? 
Yes, often 

Yes, but only once in a while 

No, hardly ever 






. 1 

. 2 
. 3 



EJ 

EJ 
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19, How often do you encourage your child to do well in school? 

Very often 1 

Somewhat often , 2 

Not too often 3 



20. How much schooling would you like to see the child who brought this questionnaire 

home complete? — " " 

Some grade school (less than 8 years) 1 

Complete grade school 2 

Some high school 3 

Complete high school 4 

Some vocational or business school after high school 5 

Complete vocational or business school after high school 6 

Some college 7 

Complete college 8 

Professional or graduate school . . 9 

21. How much schooling do you think your child will really get? 

Some grade school (less than 8 years) 1 

Complete grade school 2 

Some high school 3 

Complete high school . . 4 

Some vocational or bittiness school after high school , 5 

Complete vocational or business school after high school , , 6 

S o m e college ,••••■.. ? 

Complete college 8 

Professional or graduate school 9 






THE FOLLOWING QUESTIONS ASK ABOUT YOUR FAMILY'S EDUCATION, JOBS, INCOME, AND OTHER FACTS 

22, How much schooling have you and your husband or wife completed? 

Please circle the answers. 

Yourself Husband or Wife 



Some grade school (less than 8 years) 1 1 

Completed grade school . 2 2 

Some high school 3 3 

Completed high school . . . 4 4 

Some vocational or business school after high school , , , 5 5 

Completed vocational or business school after high school 6 6 

Some college 7 7 

Completed college 8 8 

Graduate or professional school 9 9 



23. Please circle what grades all of your children are enrolled in. 



1st grade , 


. , 1 


4th grade . „ . 4 


7th grade , 


..7 


1 0th grade .,,10 


Technical 


2nd grade 


.,2 


5th grade ... 5 


8th grade , 


. ,8 


11th grade ... 1 1 


School. . . 13 


3rd grade 


. .3 


6th grade ... 6 


9th grade , 


, , 9 


12th grade ... 12' 


College ... 14 



24, Have any of your children dropped out of school before graduating from high school? 

T es 1 INI o 2 



25, 



Do you have 
Yes . . . 



any children who 



have been in college? 

. . . 1 No. . 



2 



26, Is the husband or wife of the household currently employed? 

Husband Wife 

Yes, Full-Time 1 1 

Yes, Part-Time 2 2 

No 3 3 

Not living in house or dead 4 4 




ZZ7 
E 7 



/ * 



33, 



34, 



Jaw ui iwiiy ucjU iw>w* 

Husband's job (if any) 
Wife's job (if any) 



28, From the list below, which category best describes husband's job, wife's job? 

Please circle the one closest answer for husband; then wife. 

Husband Wife 

Mainly unskilled work (examples: cafeteria or laundry work 

unskilled factory work, car washer) 1 1 

Farmer's helper or farms other persons land 2 ■ 2 

Mainly semiskilled work (factory machine operator, gas 

station worker, delivery man, foundry worker) 3 3 

Skilled manual work (sheetmetal worker, appliance repair- 
man, railroad switchman, welder, electrician) 4 4 

Clerical or sales worker (retail store, department store, 

typist, post office, bank clerk) 5 5 

Small business owner (car dealer, gas station, dry cleaning 

shop, small grocery store) 6 6 

Professional (social worker, nurse, librarian, teacher) .... 7 7 

Large business owner or manager (branch manager of 

grocery chain, business executive, etc.) 8 8 

Professional (chemist, dentist, physician, lawyer) 9 9 

29, How many of your children are living at home? 



30, I ncluding yourself , how many people live in your house and are what you 
call family? 



Please circle the total family income for last year. Include all sources of Income 
(work, gifts, welfare, social security and so on). 



Under $2000 

Between $2000 and $2999 
Between $3000 and $3999 
Between $4000 and $4999 
Between $5000 and $5999 
Between $6000 and $6999 



1 Between $7000 and $7999 . . 

2 Between $8000 and $8999 , . 

3 Between $9000 and $9999 . . 

4 Between $10,000 and $12,999 

5 Between $1 3,000 and $1 5,999 

6 Over $16,000 



.10 

,11 

.12 



If you had no money coming into the house, how long would your savings last? 

Less than 1 week 1 1 month to 3 months , . . . 

1 week to 1 month 2 3 months to 6 months. . . . 



Have you or are you now receiving financial aid from county, state, or federal 
social welfare agencies? ' 

Yes, I am now receiving aid 1 

Yes, in the past but not now 2 
No, never 3 



35. Please circle your family position. 

Father 1 Foster parent . 

Mother 2 Family relative. 



i * • » 



Other 



. 5 



Please circle your family background. 



nencan Indian , , 
Black or Negro , . , 
Mexlcan-American , 



1 



Eskimo-Aleut 
Oriental . . . 
Puerto Rican . 



White or Caucasian 7 
Other (Please write in) 



**3 Please circle your sex, 
ERIC Male 



Female 

17g 



ZZ7 

O 



31, Please circle whether or 
Television set. , . 


not you have the following things in your home. 
Yes No 

1 2 Telephone . 


Yes 
, 1 


No 

2 


Radio 


1 


2 


Magazines . , 


, 1 


2 


Daily newspaper , 


1 


2 


Dictionary ......... 


. 1 


2 


Washing machine 


i 

i i t ( f * 


2 


Encyclopedia 


. 1 


2 


Clothes dryer . . 


1 1 ■ ■ 1 1 


2 


Children's books on . . . 
Science and Nature . . . 


’ 1 


2 



O 

EJ 

o 

ZZ7 

ZZ7 

o 



O 



co 



Ed 



K 



CO 



I 

§ 

H 




Jacks on- j 
ville I 




© 


o 

o 

fH 
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o 


I O 
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fH 
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o 


100 


"O 
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i 


! 


l 


I 


O 














lw 














■M 














M 




t—l 


VD 


cn 


o 


© 


cd 


w 




CO 


fH 




© 


W 










rH 






p- 


CM 


iH O 


© 


no 


o 


00 


rH 






© 


G 












rH 


o 














| 






rH 


CM 


© 


1 ° 


cd 


w 


m 






1 


o 


® 












! r~* 






vD 


r- 


vD 




1 ° 


co 


o 


in 


cn 






o 


'O T? 










1 


1 r4 


G *ri 














td p. 














u cd 




r-* 


rH 


ON 


cn 


1 O 


W erf 


w 




<E 






o 














I ^ 






cn 


en 


cn 


rHl 


1 O 


O 


© 






n 




O ! 


G 












I fH 
















Q) 














u 




OS 


rH 


co 


CMl 


1 €3 


Pm 


w 


CM 


fh 


in 




O 














| fH 






O 


00 


CM 


°l 


1 o 


m 


o 




ON 






o 


cd 












1 ^ 


rH 














rH 














cd 




O 


o 


o 


°i 


i o 


a 


w 




© 






o 








fH 






tH 






CM 


nO 


o 


CMl 


i © 




u 






m 




o 














; ih 


a 














o 














o 




00 


Cn] 


CM 


col 


o 


m 


w 






<E 




© 














rH 






ON 


rH 


o 


© 


o 


CQ 


u 


m 








o 


G 












rH 


cu 




























4J 




p^. 


cn 


o 


©1 


O 




m 


cn 


VD 




1 


O 












* 


rH 






tH 


o 


rH 


co 


© 


i 


u 


ON 








© 














rH 


O QJ 














^ 51 


\ 












O cd 






oo 


CM 


vD| 


o 


o 


m 


m 


fH 




CM 


© 


<c 










* 


rH 










60 

ri 












* 


i=N 




























m 














QJ 








OJ 






CL 








u 






co 






















ed 










* 










co 










OJ 




*H 










-P 


u 


G 


OJ 








•H 


m 


id 










j£* 


rH 


O, 


4J 








5 


m 


CO 


O 





co in 

O ON 

m 



cn os 
os 

m 



cn vo 
cm 

CM 



CM 

fH CJV 
vO 



cn vD 
o © 
m 



oo r- 
vo on 



O 00 
CM ON 

m 



cm m 
o © 
m 



cn © 

rH © 

in 



CTv ON 



© 
tH © 

m 



r*- r- 

© MO 

cn 



VD VD 

cn © 



Os <$- 

in © 



OS 

00 © 
nr 



vD P*- 

in cn 
cn 



in cn 

tH © 

cn 



0) 

** 4J 
<U Id 

M p4 

C/3 01 

10 

QJ G 

rH O 
CL CL 

a €0 

W QJ 

w pd 









CM 


r^ 


tH 


© 


© f 




ed 


© 


n 


<E 






© 




4J 












rH 




















•d 
















a 




© 


00 


CM 


© 


1 ° 






w 




n 






O 










, 


I rH 








i 


i 


1 


1 


I 






© 


i 


i 


i 


i 


1 




4-J 
















IW 
















cd 
















H 




rH 


CM 




© 


1 ° 






w 






Os 




O 




• 












1 fH 








CM 


00 


o 


© 


i o 




M 


© 


ON 








o 




OJ 

‘S 

fH 












1 H 


















Oi 




00 


CM 


o 


© 


1 O 




co 


m 


oo 


rH 






o 
















1 rH 








oo 




© 


n 


O 




Q) 


o 


00 








© 




rH 












tH 




H 
















4J 
















cd 




rH 


o 


© 


ON 


1 ° 




0i 


w 


© 


cn 






o 




co 
















no 




f 


i 


I 


s | 


1 ° 1 




d 


© 


1 


i 


1 


i 


o 1 




cd 












1 rH 




fH 
































o 




© 


© 


© 


oj 


l 0 




o 


m 


o 








© 








rH 








I tH 








00 


CM 


© 


oj 


i O 




d 


© 


OS 








© 




cd 












| rH 




tH 
















4-J 
















U 




00 


CM 


© 


©1 


1 O 




Q 


w 


Os 








1 o 




PH 












' fH 








1 


1 


I 


s j 


1 


1 


cd 


© 


s 


1 


i 


1 


1 




H 














rH 


rd 














H 


a 
















rH 




o 


O 


o 


©I 


o 




0) 






© 






© 




nO 






rH 




1 


rH 
























On 


tH 


© 


o 


© 




rg 


© 


-d- 


in 






© 




a 












fH 




o 
















© 












— - — 




o 




© 


-o 


© 


o| 


o 




S 


m 




OS 




1 


o 












I 


fH 










VD 


m 




© 






© 




<E 






© 




M 












rH 


Ui 


cd 
















w 
















OJ 






in 


ON 


CMl 


© 




> 


w 










© 








* 






1 


rH 



OJ 

u 

m 



tS) 







®H 






# « 






M 






QJ 






cd 




« • 


4J 






€U 


* 


0 ) 


cd 






Dh 






cd 






oo 




•H 












C/D 


0i 












w 






10 




dJ 


C 


OJ 




*H 


M 


rH 


o 


4J 


O 


d 


d) 


CL 


Oh 


®H 


cd 


cd 




a 


tn 




rH 


Oh 


4-1 


cd 


QJ 


S 


m 


oo 


O 


w 


od 



un iH 

CM Os 

<t 



m o 
© o 
in fH 



l i 



m os 
VD Os 

cn 



Os 

VD Os 
CM 



© r ^ 

VD CTs 

in 



p- vo 

iH p** 



cn rv 

00 On 



i i 
i i 



© ON 

p- os 
in 



vo * — i 
cn on 
in 



CM ON 
O ON 

in 



i i 
i i 



oo in 

p- on 

<r 



VD 00 
O ON 

cn 



CM ON 

cn on 



CM 

ON 

CM 



CM ON 
m on 



O 

ERIC 



1 

A 

£ 

3 

7 

8 

9 

1 

2 

3 

7 

8 

9 

1 

2 

3 

7 

8 

9 

c 



APPENDIX E 



TABLE E-l 



DIAN TOTAL FAMILY INCCfcffi BY SITE, GROUP, AND GRADE, 



Anchorage 


Athens 
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APPENDIX G 

TABLE G-l. PERCENTAGE DISTRIBUTION OF 'FATHER'S OCCUPATION,, FOR FULL -YEAR STUDENTS, 
BY SITE AND GROUP 
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TABLE H-l . ME' AN GEQ VALUES OF FULL-YEAR STUDENT'S FOR READING, BY 
SITE, GROUP, AND GRADE 
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TABLE H-2. MEAN GEQ VALUES OF FULL-YEAR STUDENTS FOR MATH BY 
SITE, GROUP, AND GRADE. 
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TABLE H-3. PERCENTAGE OF FULL -YEAR STUDENT'S BELOW GRADE LEVEL FOR READING, BY SITE, GROUP, AND' GRADE 
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TABLE H-4. PERCENTAGE OF FULL -YEAR STUDENTS BELOW GRADE LEVEL FOR MATHEMATICS, BY SITE, GROUP 
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TABLE 1-1. PERCENTAGE DISTRIBUTION OF RESPONSES TO ITEM 10 

(APPROVE NEW METHOD) OF PARENTS OF FULL -YEAR STUDENTS 
BY SITE AND' GROUP 
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